Jul 17 17:56:44.444526 ip-10-0-132-62 systemd[1]: Starting Kubernetes Kubelet... Jul 17 17:56:44.965505 ip-10-0-132-62 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 17:56:44.965505 ip-10-0-132-62 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 17 17:56:44.965505 ip-10-0-132-62 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 17:56:44.965505 ip-10-0-132-62 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 17 17:56:44.965505 ip-10-0-132-62 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 17:56:44.968055 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.967882 2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 17 17:56:44.977796 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977776 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:56:44.977796 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977794 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:56:44.977796 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977798 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977802 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977806 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977809 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977812 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977815 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977819 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977822 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977825 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977828 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977831 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977836 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977840 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977843 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977846 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977849 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977853 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977856 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977859 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977862 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:56:44.977893 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977864 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977868 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977871 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977874 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977876 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977879 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977882 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977884 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977888 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977892 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977897 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977900 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977903 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977907 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977910 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977913 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977915 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977919 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977922 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977925 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:56:44.978429 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977928 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977930 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977933 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977936 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977939 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977941 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977944 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977947 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977949 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977952 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977955 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977958 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977961 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977972 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977976 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977979 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977982 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977984 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977987 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977990 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:56:44.978933 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977993 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977996 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.977998 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978001 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978005 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978007 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978010 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978013 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978016 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978019 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978022 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978025 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978030 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978034 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978037 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978040 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978044 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978047 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978049 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978052 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:56:44.979447 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978055 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978057 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978060 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978063 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978066 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978068 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978072 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978075 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978078 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978081 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978095 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978097 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978102 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978106 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978109 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978112 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.978115 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979725 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979736 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:56:44.979946 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979740 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979744 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979747 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979750 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979753 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979756 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979759 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979762 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979764 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979769 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979772 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979775 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979778 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979781 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979784 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979787 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979789 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979792 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979795 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979799 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:56:44.980555 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979802 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979805 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979807 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979812 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979816 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979819 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979822 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979825 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979828 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979831 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979834 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979836 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979839 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979842 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979844 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979847 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979850 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979853 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979856 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:56:44.981127 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979859 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979862 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979864 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979867 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979870 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979872 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979875 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979878 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979880 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979883 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979887 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979890 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979895 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979898 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979901 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979904 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979907 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979910 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979913 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:56:44.981613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979916 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979918 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979921 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979925 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979927 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979930 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979933 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979936 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979938 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979941 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979944 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979947 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979950 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979953 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979956 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979960 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979963 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979966 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979969 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979972 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:56:44.982113 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979975 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979977 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979980 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979983 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979987 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979990 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979993 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.979995 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980000 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980003 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980005 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980008 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980011 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980013 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980016 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980019 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980022 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980025 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980027 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:56:44.982615 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980456 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980461 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980465 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980480 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980484 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980489 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980493 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980497 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980500 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980503 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980506 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980509 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980513 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980516 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980519 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980522 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980524 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980529 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980533 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:56:44.983103 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980538 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980542 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980545 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980548 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980551 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980554 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980558 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980561 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980564 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980567 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980570 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980573 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980576 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980579 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980581 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980584 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980587 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980590 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980593 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:56:44.983608 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980597 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980599 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980602 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980605 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980607 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980610 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980613 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980615 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980618 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980621 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980623 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980627 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980629 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980632 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980635 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980638 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980641 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980643 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980646 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980649 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:56:44.984112 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980652 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980655 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980657 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980660 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980663 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980666 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980668 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980671 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980673 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980676 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980679 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980682 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980684 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980687 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980690 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980692 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980695 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980698 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980700 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980703 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980706 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:56:44.984618 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980708 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980711 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980715 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980718 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980721 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980723 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980727 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980730 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980732 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980735 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980738 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980741 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980744 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980746 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980749 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980752 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980755 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980758 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980762 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.980765 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:56:44.985177 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980846 2572 flags.go:64] FLAG: --address="0.0.0.0" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980864 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980875 2572 flags.go:64] FLAG: --anonymous-auth="true" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980883 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980888 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980893 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980898 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980903 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980906 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980909 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980913 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980917 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980920 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980924 2572 flags.go:64] FLAG: --cgroup-root="" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980927 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980930 2572 flags.go:64] FLAG: --client-ca-file="" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980933 2572 flags.go:64] FLAG: --cloud-provider="external" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980937 2572 flags.go:64] FLAG: --cluster-dns="[]" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980941 2572 flags.go:64] FLAG: --cluster-domain="" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980944 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980947 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980951 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980954 2572 flags.go:64] FLAG: --container-log-max-files="5" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980959 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 17 17:56:44.985672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980962 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980966 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980969 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980972 2572 flags.go:64] FLAG: --contention-profiling="false" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980975 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980978 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980982 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980985 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980989 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980993 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980996 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.980999 2572 flags.go:64] FLAG: --enable-load-reader="false" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981003 2572 flags.go:64] FLAG: --enable-server="true" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981006 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981011 2572 flags.go:64] FLAG: --event-burst="100" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981015 2572 flags.go:64] FLAG: --event-qps="50" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981018 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981021 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981025 2572 flags.go:64] FLAG: --eviction-hard="" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981029 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981032 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981035 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981039 2572 flags.go:64] FLAG: --eviction-soft="" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981044 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981047 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 17 17:56:44.986287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981051 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981054 2572 flags.go:64] FLAG: --experimental-mounter-path="" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981057 2572 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981060 2572 flags.go:64] FLAG: --fail-swap-on="true" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981063 2572 flags.go:64] FLAG: --feature-gates="" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981067 2572 flags.go:64] FLAG: --file-check-frequency="20s" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981071 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981074 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981077 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981081 2572 flags.go:64] FLAG: --healthz-port="10248" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981098 2572 flags.go:64] FLAG: --help="false" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981101 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-132-62.ec2.internal" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981105 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981109 2572 flags.go:64] FLAG: --http-check-frequency="20s" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981112 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981115 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981119 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981122 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981125 2572 flags.go:64] FLAG: --image-service-endpoint="" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981128 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981131 2572 flags.go:64] FLAG: --kube-api-burst="100" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981135 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981139 2572 flags.go:64] FLAG: --kube-api-qps="50" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981142 2572 flags.go:64] FLAG: --kube-reserved="" Jul 17 17:56:44.986964 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981145 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981149 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981153 2572 flags.go:64] FLAG: --kubelet-cgroups="" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981156 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981159 2572 flags.go:64] FLAG: --lock-file="" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981162 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981167 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981170 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981176 2572 flags.go:64] FLAG: --log-json-split-stream="false" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981179 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981183 2572 flags.go:64] FLAG: --log-text-split-stream="false" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981186 2572 flags.go:64] FLAG: --logging-format="text" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981189 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981192 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981196 2572 flags.go:64] FLAG: --manifest-url="" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981199 2572 flags.go:64] FLAG: --manifest-url-header="" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981203 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981207 2572 flags.go:64] FLAG: --max-open-files="1000000" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981211 2572 flags.go:64] FLAG: --max-pods="110" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981215 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981218 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981221 2572 flags.go:64] FLAG: --memory-manager-policy="None" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981224 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981228 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981231 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 17 17:56:44.987656 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981234 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981242 2572 flags.go:64] FLAG: --node-status-max-images="50" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981245 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981248 2572 flags.go:64] FLAG: --oom-score-adj="-999" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981251 2572 flags.go:64] FLAG: --pod-cidr="" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981254 2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981261 2572 flags.go:64] FLAG: --pod-manifest-path="" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981264 2572 flags.go:64] FLAG: --pod-max-pids="-1" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981268 2572 flags.go:64] FLAG: --pods-per-core="0" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981271 2572 flags.go:64] FLAG: --port="10250" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981274 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981277 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fae46eccd24472a8" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981281 2572 flags.go:64] FLAG: --qos-reserved="" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981286 2572 flags.go:64] FLAG: --read-only-port="10255" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981289 2572 flags.go:64] FLAG: --register-node="true" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981292 2572 flags.go:64] FLAG: --register-with-taints="" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981296 2572 flags.go:64] FLAG: --registry-burst="10" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981299 2572 flags.go:64] FLAG: --registry-qps="5" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981302 2572 flags.go:64] FLAG: --reserved-cpus="" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981305 2572 flags.go:64] FLAG: --reserved-memory="" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981309 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981312 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981316 2572 flags.go:64] FLAG: --rotate-certificates="false" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981319 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981322 2572 flags.go:64] FLAG: --runonce="false" Jul 17 17:56:44.988356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981325 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981328 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981332 2572 flags.go:64] FLAG: --seccomp-default="false" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981335 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981339 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981342 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981346 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981349 2572 flags.go:64] FLAG: --storage-driver-password="root" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981352 2572 flags.go:64] FLAG: --storage-driver-secure="false" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981355 2572 flags.go:64] FLAG: --storage-driver-table="stats" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981359 2572 flags.go:64] FLAG: --storage-driver-user="root" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981362 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981366 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981369 2572 flags.go:64] FLAG: --system-cgroups="" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981373 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981378 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981382 2572 flags.go:64] FLAG: --tls-cert-file="" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981385 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981388 2572 flags.go:64] FLAG: --tls-min-version="" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981391 2572 flags.go:64] FLAG: --tls-private-key-file="" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981396 2572 flags.go:64] FLAG: --topology-manager-policy="none" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981399 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981403 2572 flags.go:64] FLAG: --topology-manager-scope="container" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981406 2572 flags.go:64] FLAG: --v="2" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981410 2572 flags.go:64] FLAG: --version="false" Jul 17 17:56:44.989440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981414 2572 flags.go:64] FLAG: --vmodule="" Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981419 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.981422 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981548 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981551 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981555 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981558 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981561 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981565 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981568 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981571 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981575 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981578 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981581 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981583 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981586 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981589 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981592 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981595 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981597 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:56:44.990477 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981601 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981604 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981607 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981610 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981613 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981616 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981619 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981624 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981627 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981631 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981635 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981638 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981641 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981644 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981648 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981652 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981655 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981658 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981661 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:56:44.991015 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981664 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981667 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981670 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981673 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981677 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981679 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981683 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981685 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981689 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981691 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981695 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981698 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981700 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981703 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981706 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981709 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981713 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981715 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981718 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981721 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:56:44.991556 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981725 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981728 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981731 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981734 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981736 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981739 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981742 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981746 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981750 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981753 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981756 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981759 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981762 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981765 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981768 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981771 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981775 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981778 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981781 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:56:44.992080 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981784 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981786 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981789 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981792 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981795 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981797 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981800 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981803 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981806 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981809 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981812 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981815 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981818 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981822 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981825 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981828 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981830 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981834 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981837 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981839 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:56:44.992660 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981842 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:56:44.993174 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981845 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:56:44.993174 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981847 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:56:44.993174 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.981850 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:56:44.993174 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.982747 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 17:56:44.993174 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.991339 2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 17 17:56:44.993174 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.991357 2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 17 17:56:44.993174 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.991376 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 17 17:56:44.993174 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.991386 2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 17 17:56:44.993174 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991449 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:56:44.993174 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991453 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:56:44.993174 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991457 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:56:44.993174 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991461 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:56:44.993174 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991464 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:56:44.993174 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991467 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:56:44.993174 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991470 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991473 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991476 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991479 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991482 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991485 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991488 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991491 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991493 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991496 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991499 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991502 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991504 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991507 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991510 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991512 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991515 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991518 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991521 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:56:44.993565 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991524 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991527 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991529 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991533 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991537 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991540 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991543 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991545 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991549 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991551 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991554 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991557 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991560 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991563 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991566 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991569 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991572 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991575 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991578 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991580 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:56:44.994039 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991583 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991585 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991588 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991591 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991595 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991599 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991603 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991605 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991608 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991611 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991614 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991617 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991620 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991623 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991625 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991629 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991632 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991635 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991639 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:56:44.994558 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991643 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991646 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991650 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991652 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991655 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991657 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991660 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991663 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991666 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991669 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991672 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991675 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991678 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991681 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991684 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991687 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991691 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991695 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991699 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:56:44.995041 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991702 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:56:44.995544 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991705 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:56:44.995544 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991708 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:56:44.995544 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991710 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:56:44.995544 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991713 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:56:44.995544 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991716 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:56:44.995544 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991719 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:56:44.995544 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991721 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:56:44.995544 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991724 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:56:44.995544 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991727 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:56:44.995544 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991730 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:56:44.995544 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991733 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:56:44.995544 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991736 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:56:44.995544 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991739 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:56:44.995544 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991742 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:56:44.995544 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991744 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:56:44.995929 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.991750 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 17:56:44.995929 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991891 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:56:44.995929 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991897 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:56:44.995929 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991901 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:56:44.995929 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991904 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:56:44.995929 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991907 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:56:44.995929 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991910 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:56:44.995929 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991913 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:56:44.995929 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991916 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:56:44.995929 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991919 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:56:44.995929 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991921 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:56:44.995929 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991924 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:56:44.995929 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991927 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:56:44.995929 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991930 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:56:44.995929 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991933 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991936 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991939 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991942 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991946 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991951 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991954 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991957 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991960 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991962 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991965 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991968 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991971 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991975 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991978 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991981 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991983 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991986 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991989 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:56:44.996420 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991992 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991995 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.991997 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992000 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992003 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992005 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992008 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992010 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992013 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992016 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992019 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992021 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992024 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992027 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992030 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992033 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992036 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992038 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992042 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992044 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:56:44.996892 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992047 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992050 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992054 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992057 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992061 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992064 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992068 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992072 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992075 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992078 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992102 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992106 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992109 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992112 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992115 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992117 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992120 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992123 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992126 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:56:44.997399 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992129 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992131 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992134 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992137 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992140 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992143 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992145 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992150 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992153 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992156 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992159 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992161 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992164 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992167 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992170 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992173 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992176 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992179 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992182 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992185 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:56:44.997877 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992188 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:56:44.998395 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992190 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:56:44.998395 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992193 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:56:44.998395 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992196 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:56:44.998395 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992199 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:56:44.998395 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992201 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:56:44.998395 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992204 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:56:44.998395 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:44.992207 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:56:44.998395 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.992212 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 17:56:44.998395 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.992343 2572 server.go:962] "Client rotation is on, will bootstrap in background" Jul 17 17:56:44.998395 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.995219 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 17 17:56:44.998395 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.996596 2572 server.go:1019] "Starting client certificate rotation" Jul 17 17:56:44.998395 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.996706 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 17:56:44.998395 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.996740 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 17:56:44.999576 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:44.999561 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 17:56:45.003383 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.003367 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 17:56:45.016640 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.016616 2572 log.go:25] "Validated CRI v1 runtime API" Jul 17 17:56:45.018707 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.018689 2572 log.go:25] "Validated CRI v1 image API" Jul 17 17:56:45.019570 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.019550 2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 17 17:56:45.026825 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.026805 2572 fs.go:135] Filesystem UUIDs: map[2da00920-eaae-4846-b74e-3afaddcf6e24:/dev/nvme0n1p4 3afb3dc0-0cdb-4365-8c83-2cf9532cad64:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 17 17:56:45.026894 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.026825 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 17 17:56:45.029774 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.029756 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 17:56:45.033791 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.033634 2572 manager.go:216] Machine: {Timestamp:2026-07-17 17:56:45.033109549 +0000 UTC m=+0.442537172 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102141 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c3f68963c4921bcf6a8f0bdc6439f SystemUUID:ec2c3f68-963c-4921-bcf6-a8f0bdc6439f BootID:882fe8f3-8ee9-499e-9a20-be3f9d368263 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cd:d3:86:b0:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cd:d3:86:b0:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:bb:d4:bc:a8:ae Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 17 17:56:45.033791 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.033779 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 17 17:56:45.033976 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.033900 2572 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 17 17:56:45.041906 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.041872 2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 17 17:56:45.042049 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.041906 2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-62.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 17 17:56:45.043102 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.043082 2572 topology_manager.go:138] "Creating topology manager with none policy" Jul 17 17:56:45.043102 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.043096 2572 container_manager_linux.go:309] "Creating device plugin manager" Jul 17 17:56:45.043174 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.043104 2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 17:56:45.043174 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.043116 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 17:56:45.044721 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.044709 2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 17 17:56:45.048675 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.048659 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 17 17:56:45.048805 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.048794 2572 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 17 17:56:45.048848 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.048829 2572 kubelet.go:486] "Attempting to sync node with API server" Jul 17 17:56:45.048848 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.048841 2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 17 17:56:45.048946 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.048859 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 17 17:56:45.048946 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.048879 2572 kubelet.go:398] "Adding apiserver pod source" Jul 17 17:56:45.048946 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.048889 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 17 17:56:45.049313 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.049300 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 17:56:45.049370 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.049319 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 17:56:45.052688 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.052667 2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 17 17:56:45.057470 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.057453 2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 17 17:56:45.057554 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.057486 2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 17 17:56:45.058741 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.058722 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 17 17:56:45.058894 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.058870 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-62.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 17 17:56:45.059851 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.059837 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 17 17:56:45.059908 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.059856 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 17 17:56:45.059908 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.059864 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 17 17:56:45.059908 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.059871 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 17 17:56:45.059908 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.059884 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 17 17:56:45.059908 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.059891 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 17 17:56:45.059908 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.059899 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 17 17:56:45.059908 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.059906 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 17 17:56:45.059908 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.059913 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 17 17:56:45.060142 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.059920 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 17 17:56:45.060142 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.059929 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 17 17:56:45.060142 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.059935 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 17 17:56:45.061610 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.061600 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 17 17:56:45.061646 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.061612 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 17 17:56:45.061676 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.061650 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xhtw7" Jul 17 17:56:45.065048 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.065029 2572 server.go:1268] "Started kubelet" Jul 17 17:56:45.065174 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.065120 2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 17 17:56:45.065224 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.065134 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 17 17:56:45.065224 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.065213 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 17 17:56:45.065996 ip-10-0-132-62 systemd[1]: Started Kubernetes Kubelet. Jul 17 17:56:45.068122 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.068106 2572 server.go:310] "Adding debug handlers to kubelet server" Jul 17 17:56:45.070021 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.069995 2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 17 17:56:45.070702 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.070683 2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 17 17:56:45.072103 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.072070 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-62.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 17 17:56:45.072448 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.072430 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xhtw7" Jul 17 17:56:45.073129 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.072145 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-62.ec2.internal.18c32567c9b82708 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-62.ec2.internal,UID:ip-10-0-132-62.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-62.ec2.internal,},FirstTimestamp:2026-07-17 17:56:45.064996616 +0000 UTC m=+0.474424216,LastTimestamp:2026-07-17 17:56:45.064996616 +0000 UTC m=+0.474424216,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-62.ec2.internal,}" Jul 17 17:56:45.078836 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.078813 2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 17 17:56:45.079017 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.079000 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-62.ec2.internal\" not found" Jul 17 17:56:45.079133 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.078819 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 17 17:56:45.079512 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.079491 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 17 17:56:45.079589 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.079550 2572 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 17 17:56:45.079589 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.079574 2572 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 17 17:56:45.079755 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.079726 2572 reconstruct.go:97] "Volume reconstruction finished" Jul 17 17:56:45.079755 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.079739 2572 reconciler.go:29] "Reconciler: start to sync state" Jul 17 17:56:45.079966 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.079944 2572 factory.go:55] Registering systemd factory Jul 17 17:56:45.079966 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.079966 2572 factory.go:223] Registration of the systemd container factory successfully Jul 17 17:56:45.080246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.080229 2572 factory.go:153] Registering CRI-O factory Jul 17 17:56:45.080246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.080249 2572 factory.go:223] Registration of the crio container factory successfully Jul 17 17:56:45.080396 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.080306 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 17 17:56:45.080396 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.080333 2572 factory.go:103] Registering Raw factory Jul 17 17:56:45.080396 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.080350 2572 manager.go:1196] Started watching for new ooms in manager Jul 17 17:56:45.081820 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.081796 2572 manager.go:349] Starting recovery of all containers Jul 17 17:56:45.081914 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.081889 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 17 17:56:45.081982 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.081900 2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-62.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 17 17:56:45.093542 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.093396 2572 manager.go:354] Recovery completed Jul 17 17:56:45.098045 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.098026 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:56:45.100564 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.100547 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:56:45.100661 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.100584 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:56:45.100661 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.100601 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:56:45.101229 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.101212 2572 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 17 17:56:45.101229 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.101229 2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 17 17:56:45.101370 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.101244 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:56:45.101370 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.101246 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 17 17:56:45.102154 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.102135 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:56:45.102249 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.102170 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:56:45.102249 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.102185 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:56:45.102789 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.102777 2572 policy_none.go:49] "None policy: Start" Jul 17 17:56:45.102841 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.102801 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 17 17:56:45.102841 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.102816 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 17 17:56:45.104108 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.104094 2572 policy_none.go:47] "Start" Jul 17 17:56:45.147753 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.147735 2572 manager.go:337] "Starting Device Plugin manager" Jul 17 17:56:45.147842 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.147776 2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 17 17:56:45.147842 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.147797 2572 server.go:87] "Starting device plugin registration server" Jul 17 17:56:45.148101 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.148086 2572 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 17 17:56:45.148149 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.148100 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 17 17:56:45.148302 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.148186 2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 17 17:56:45.148302 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.148290 2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 17 17:56:45.148302 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.148302 2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 17 17:56:45.148863 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.148843 2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 17 17:56:45.148940 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.148900 2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-62.ec2.internal\" not found" Jul 17 17:56:45.217552 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.217487 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 17 17:56:45.218782 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.218760 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 17 17:56:45.218843 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.218813 2572 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 17 17:56:45.218843 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.218841 2572 kubelet.go:2448] "Starting kubelet main sync loop" Jul 17 17:56:45.218934 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.218884 2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 17 17:56:45.221344 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.221323 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:56:45.248809 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.248777 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:56:45.251024 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.250994 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:56:45.251103 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.251047 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:56:45.251103 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.251062 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:56:45.251103 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.251088 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-62.ec2.internal" Jul 17 17:56:45.259402 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.259386 2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-62.ec2.internal" Jul 17 17:56:45.259456 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.259412 2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-62.ec2.internal\": node \"ip-10-0-132-62.ec2.internal\" not found" Jul 17 17:56:45.276428 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.276398 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-62.ec2.internal\" not found" Jul 17 17:56:45.319933 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.319902 2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-62.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal"] Jul 17 17:56:45.320069 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.319978 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:56:45.321102 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.321074 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:56:45.321159 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.321103 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:56:45.321159 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.321120 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:56:45.322576 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.322564 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:56:45.322738 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.322711 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-62.ec2.internal" Jul 17 17:56:45.322790 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.322742 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:56:45.323680 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.323663 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:56:45.323779 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.323696 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:56:45.323779 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.323701 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:56:45.323779 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.323711 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:56:45.323779 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.323722 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:56:45.323779 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.323732 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:56:45.325765 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.325746 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal" Jul 17 17:56:45.325860 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.325779 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:56:45.327975 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.327955 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:56:45.328078 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.327985 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:56:45.328078 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.327996 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:56:45.357795 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.357774 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-62.ec2.internal\" not found" node="ip-10-0-132-62.ec2.internal" Jul 17 17:56:45.362546 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.362527 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-62.ec2.internal\" not found" node="ip-10-0-132-62.ec2.internal" Jul 17 17:56:45.376747 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.376727 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-62.ec2.internal\" not found" Jul 17 17:56:45.381270 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.381252 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b28b739fa6bf195253cfa6375ca90e6-config\") pod \"kube-apiserver-proxy-ip-10-0-132-62.ec2.internal\" (UID: \"8b28b739fa6bf195253cfa6375ca90e6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-62.ec2.internal" Jul 17 17:56:45.381356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.381284 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/286e9012d20d5d00cdc3b7cfc23fe73f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal\" (UID: \"286e9012d20d5d00cdc3b7cfc23fe73f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal" Jul 17 17:56:45.381356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.381317 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/286e9012d20d5d00cdc3b7cfc23fe73f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal\" (UID: \"286e9012d20d5d00cdc3b7cfc23fe73f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal" Jul 17 17:56:45.477692 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.477597 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-62.ec2.internal\" not found" Jul 17 17:56:45.482282 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.482262 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/286e9012d20d5d00cdc3b7cfc23fe73f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal\" (UID: \"286e9012d20d5d00cdc3b7cfc23fe73f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal" Jul 17 17:56:45.482347 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.482294 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b28b739fa6bf195253cfa6375ca90e6-config\") pod \"kube-apiserver-proxy-ip-10-0-132-62.ec2.internal\" (UID: \"8b28b739fa6bf195253cfa6375ca90e6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-62.ec2.internal" Jul 17 17:56:45.482347 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.482310 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/286e9012d20d5d00cdc3b7cfc23fe73f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal\" (UID: \"286e9012d20d5d00cdc3b7cfc23fe73f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal" Jul 17 17:56:45.482528 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.482367 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8b28b739fa6bf195253cfa6375ca90e6-config\") pod \"kube-apiserver-proxy-ip-10-0-132-62.ec2.internal\" (UID: \"8b28b739fa6bf195253cfa6375ca90e6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-62.ec2.internal" Jul 17 17:56:45.482528 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.482380 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/286e9012d20d5d00cdc3b7cfc23fe73f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal\" (UID: \"286e9012d20d5d00cdc3b7cfc23fe73f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal" Jul 17 17:56:45.482528 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.482449 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/286e9012d20d5d00cdc3b7cfc23fe73f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal\" (UID: \"286e9012d20d5d00cdc3b7cfc23fe73f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal" Jul 17 17:56:45.578086 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.578030 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-62.ec2.internal\" not found" Jul 17 17:56:45.667365 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.667339 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-62.ec2.internal" Jul 17 17:56:45.667453 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.667362 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal" Jul 17 17:56:45.678518 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.678490 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-62.ec2.internal\" not found" Jul 17 17:56:45.782554 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.782450 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-62.ec2.internal\" not found" Jul 17 17:56:45.886489 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.886435 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-62.ec2.internal\" not found" Jul 17 17:56:45.986749 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:45.986703 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-62.ec2.internal\" not found" Jul 17 17:56:45.996233 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.996214 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 17 17:56:45.996394 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:45.996376 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 17:56:46.065720 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:46.065631 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:56:46.078234 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:46.078181 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 17:51:45 +0000 UTC" deadline="2028-02-01 13:43:55.716278316 +0000 UTC" Jul 17 17:56:46.078234 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:46.078225 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13531h47m9.638055745s" Jul 17 17:56:46.087116 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:46.087076 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-62.ec2.internal\" not found" Jul 17 17:56:46.139112 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:46.139091 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:56:46.179475 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:46.179438 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-62.ec2.internal" Jul 17 17:56:46.188595 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:46.188573 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 17:56:46.189572 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:46.189553 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal" Jul 17 17:56:46.202046 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:46.202006 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 17:56:46.275840 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:46.275792 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286e9012d20d5d00cdc3b7cfc23fe73f.slice/crio-db91719345149e7cfd071c3b26c5fc1040924dc3c1a53371e9f12f2f63cc206d WatchSource:0}: Error finding container db91719345149e7cfd071c3b26c5fc1040924dc3c1a53371e9f12f2f63cc206d: Status 404 returned error can't find the container with id db91719345149e7cfd071c3b26c5fc1040924dc3c1a53371e9f12f2f63cc206d Jul 17 17:56:46.277613 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:46.277575 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b28b739fa6bf195253cfa6375ca90e6.slice/crio-618d8f17b14067e0ded1091dbb7c7ef88b8bb8cec531c052de3d93505e508b3a WatchSource:0}: Error finding container 618d8f17b14067e0ded1091dbb7c7ef88b8bb8cec531c052de3d93505e508b3a: Status 404 returned error can't find the container with id 618d8f17b14067e0ded1091dbb7c7ef88b8bb8cec531c052de3d93505e508b3a Jul 17 17:56:46.282047 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:46.282031 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 17:56:46.559915 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:46.559830 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:56:47.050688 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.050600 2572 apiserver.go:52] "Watching apiserver" Jul 17 17:56:47.059771 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.059738 2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 17:56:47.060272 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.060241 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-62.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2","openshift-dns/node-resolver-gpnwp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal","openshift-multus/multus-additional-cni-plugins-sz4tq","openshift-network-diagnostics/network-check-target-wcf2q","kube-system/konnectivity-agent-xbw8z","openshift-cluster-node-tuning-operator/tuned-nwszv","openshift-image-registry/node-ca-hfrgj","openshift-multus/multus-m8chw","openshift-multus/network-metrics-daemon-5ccsk","openshift-network-operator/iptables-alerter-gqftj","openshift-ovn-kubernetes/ovnkube-node-8v7vg","kube-system/global-pull-secret-syncer-ts454"] Jul 17 17:56:47.063284 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.063259 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.065445 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.065424 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.066343 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.066193 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4q6d7\"" Jul 17 17:56:47.066343 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.066234 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 17:56:47.066513 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.066469 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 17:56:47.068466 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.068229 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 17:56:47.068466 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.068358 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 17 17:56:47.068466 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.068441 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 17:56:47.068670 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.068575 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gfvzt\"" Jul 17 17:56:47.068753 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.068735 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gpnwp" Jul 17 17:56:47.071713 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.071693 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vkgj2\"" Jul 17 17:56:47.071925 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.071909 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 17:56:47.072169 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.072151 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 17:56:47.072769 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.072477 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.075129 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.075108 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 17:56:47.075376 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.075358 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 17:56:47.075562 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.075544 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 17:56:47.075754 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.075737 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 17:56:47.078628 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.077057 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hfrgj" Jul 17 17:56:47.078628 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.077079 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-756sv\"" Jul 17 17:56:47.078628 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.077297 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 17:56:47.078628 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.078296 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 17:56:47.078628 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.078502 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:56:47.078628 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.078611 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:56:47.079894 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.079736 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 17:56:47.080279 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.080158 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nk4pk\"" Jul 17 17:56:47.080279 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.080197 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 17 17:56:47.080279 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.080249 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 17:56:47.080610 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.080596 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 17:56:47.081101 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.081079 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xbw8z" Jul 17 17:56:47.084255 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.084231 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.084474 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.084455 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-csght\"" Jul 17 17:56:47.084684 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.084667 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 17:56:47.084766 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.084317 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 17:56:47.086608 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.086586 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.087078 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.087013 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 17:56:47.087182 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.087131 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 17:56:47.087250 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.087207 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 17:56:47.087572 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.087552 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 17:56:47.087720 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.087555 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 17:56:47.088249 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.087980 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7frs5\"" Jul 17 17:56:47.089273 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.089148 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 17:56:47.089591 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.089542 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9lzs9\"" Jul 17 17:56:47.090147 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.090131 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:56:47.090340 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.090317 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:56:47.092901 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.092880 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gqftj" Jul 17 17:56:47.093339 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093027 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-etc-sysctl-conf\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.093339 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093068 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rlsn\" (UniqueName: \"kubernetes.io/projected/2e007a42-9178-41c7-bd04-0544e3996bce-kube-api-access-5rlsn\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.093339 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093136 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 17:56:47.093339 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093143 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-kubelet\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.093339 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093163 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-run-systemd\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.093339 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093191 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-run-ovn\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.093339 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093213 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-etc-systemd\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.093339 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093235 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-sys\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.093339 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093259 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-var-lib-kubelet\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.093339 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093300 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e007a42-9178-41c7-bd04-0544e3996bce-tmp\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.093339 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093340 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8841f5d-e09f-4e42-b700-0002b5a3a342-host\") pod \"node-ca-hfrgj\" (UID: \"c8841f5d-e09f-4e42-b700-0002b5a3a342\") " pod="openshift-image-registry/node-ca-hfrgj" Jul 17 17:56:47.093937 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093383 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0bc2b05e-ac19-449a-b861-d729822c0afa-socket-dir\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.093937 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093402 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0bc2b05e-ac19-449a-b861-d729822c0afa-registration-dir\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.093937 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093422 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm5wk\" (UniqueName: \"kubernetes.io/projected/0bc2b05e-ac19-449a-b861-d729822c0afa-kube-api-access-lm5wk\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.093937 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093446 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-host\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.093937 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093492 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-etc-sysctl-d\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.093937 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093531 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-run\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.093937 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093570 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e007a42-9178-41c7-bd04-0544e3996bce-etc-tuned\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.093937 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093594 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-log-socket\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.093937 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093624 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a78443b7-6add-4ea0-9b24-949b0b9d117a-ovn-node-metrics-cert\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.093937 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093664 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a78443b7-6add-4ea0-9b24-949b0b9d117a-ovnkube-script-lib\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.093937 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093690 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-run-netns\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.093937 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093713 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-node-log\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.093937 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093751 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-run-ovn-kubernetes\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.093937 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093776 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.093937 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093802 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltd2r\" (UniqueName: \"kubernetes.io/projected/a78443b7-6add-4ea0-9b24-949b0b9d117a-kube-api-access-ltd2r\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.093937 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093827 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0bc2b05e-ac19-449a-b861-d729822c0afa-sys-fs\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.094758 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093848 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48aa1cde-e776-496a-9816-9a67614f0210-konnectivity-ca\") pod \"konnectivity-agent-xbw8z\" (UID: \"48aa1cde-e776-496a-9816-9a67614f0210\") " pod="kube-system/konnectivity-agent-xbw8z" Jul 17 17:56:47.094758 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093869 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-etc-modprobe-d\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.094758 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093894 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-run-openvswitch\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.094758 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093926 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-cni-bin\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.094758 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093956 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltg2b\" (UniqueName: \"kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b\") pod \"network-check-target-wcf2q\" (UID: \"27188dfc-708b-462f-9a9a-24fda6997009\") " pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:56:47.094758 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.093983 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0bc2b05e-ac19-449a-b861-d729822c0afa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.094758 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.094008 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0bc2b05e-ac19-449a-b861-d729822c0afa-etc-selinux\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.094758 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.094048 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/376644cc-94a0-41d2-9cc0-cc410806ecf8-hosts-file\") pod \"node-resolver-gpnwp\" (UID: \"376644cc-94a0-41d2-9cc0-cc410806ecf8\") " pod="openshift-dns/node-resolver-gpnwp" Jul 17 17:56:47.094758 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.094072 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-etc-sysconfig\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.094758 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.094095 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-slash\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.094758 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.094119 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a78443b7-6add-4ea0-9b24-949b0b9d117a-ovnkube-config\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.094758 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.094143 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c8841f5d-e09f-4e42-b700-0002b5a3a342-serviceca\") pod \"node-ca-hfrgj\" (UID: \"c8841f5d-e09f-4e42-b700-0002b5a3a342\") " pod="openshift-image-registry/node-ca-hfrgj" Jul 17 17:56:47.094758 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.094168 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6xln\" (UniqueName: \"kubernetes.io/projected/c8841f5d-e09f-4e42-b700-0002b5a3a342-kube-api-access-r6xln\") pod \"node-ca-hfrgj\" (UID: \"c8841f5d-e09f-4e42-b700-0002b5a3a342\") " pod="openshift-image-registry/node-ca-hfrgj" Jul 17 17:56:47.094758 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.094192 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95sst\" (UniqueName: \"kubernetes.io/projected/376644cc-94a0-41d2-9cc0-cc410806ecf8-kube-api-access-95sst\") pod \"node-resolver-gpnwp\" (UID: \"376644cc-94a0-41d2-9cc0-cc410806ecf8\") " pod="openshift-dns/node-resolver-gpnwp" Jul 17 17:56:47.094758 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.094218 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48aa1cde-e776-496a-9816-9a67614f0210-agent-certs\") pod \"konnectivity-agent-xbw8z\" (UID: \"48aa1cde-e776-496a-9816-9a67614f0210\") " pod="kube-system/konnectivity-agent-xbw8z" Jul 17 17:56:47.094758 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.094242 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-etc-kubernetes\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.095550 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.094268 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-lib-modules\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.095550 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.094292 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-etc-openvswitch\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.095550 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.094317 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0bc2b05e-ac19-449a-b861-d729822c0afa-device-dir\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.095550 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.094347 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-systemd-units\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.095550 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.094371 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-var-lib-openvswitch\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.095550 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.094395 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-cni-netd\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.095550 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.094422 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a78443b7-6add-4ea0-9b24-949b0b9d117a-env-overrides\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.095550 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.094447 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/376644cc-94a0-41d2-9cc0-cc410806ecf8-tmp-dir\") pod \"node-resolver-gpnwp\" (UID: \"376644cc-94a0-41d2-9cc0-cc410806ecf8\") " pod="openshift-dns/node-resolver-gpnwp" Jul 17 17:56:47.095550 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.095431 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:47.095550 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.095498 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:56:47.096034 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.095577 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 17:56:47.096034 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.095613 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d48g6\"" Jul 17 17:56:47.096034 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.095964 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 17:56:47.096208 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.096197 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 17:56:47.122144 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.122117 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g6v4g" Jul 17 17:56:47.132482 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.132458 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g6v4g" Jul 17 17:56:47.181913 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.181877 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 17 17:56:47.195350 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195316 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n4qp\" (UniqueName: \"kubernetes.io/projected/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-kube-api-access-9n4qp\") pod \"network-metrics-daemon-5ccsk\" (UID: \"e05c7832-1be4-46bd-91e4-2dd5c8abeb02\") " pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:56:47.195515 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195360 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-etc-systemd\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.195515 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195393 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-var-lib-kubelet\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.195515 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195471 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-var-lib-kubelet\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.195515 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195471 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-etc-systemd\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.195515 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195503 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8841f5d-e09f-4e42-b700-0002b5a3a342-host\") pod \"node-ca-hfrgj\" (UID: \"c8841f5d-e09f-4e42-b700-0002b5a3a342\") " pod="openshift-image-registry/node-ca-hfrgj" Jul 17 17:56:47.195777 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195525 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/376644cc-94a0-41d2-9cc0-cc410806ecf8-hosts-file\") pod \"node-resolver-gpnwp\" (UID: \"376644cc-94a0-41d2-9cc0-cc410806ecf8\") " pod="openshift-dns/node-resolver-gpnwp" Jul 17 17:56:47.195777 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195546 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-multus-cni-dir\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.195777 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195574 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-etc-kubernetes\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.195777 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195591 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8841f5d-e09f-4e42-b700-0002b5a3a342-host\") pod \"node-ca-hfrgj\" (UID: \"c8841f5d-e09f-4e42-b700-0002b5a3a342\") " pod="openshift-image-registry/node-ca-hfrgj" Jul 17 17:56:47.195777 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195610 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/376644cc-94a0-41d2-9cc0-cc410806ecf8-hosts-file\") pod \"node-resolver-gpnwp\" (UID: \"376644cc-94a0-41d2-9cc0-cc410806ecf8\") " pod="openshift-dns/node-resolver-gpnwp" Jul 17 17:56:47.195777 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195602 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkbwn\" (UniqueName: \"kubernetes.io/projected/fefc9465-ec24-48f7-9a99-d6fde7791695-kube-api-access-bkbwn\") pod \"iptables-alerter-gqftj\" (UID: \"fefc9465-ec24-48f7-9a99-d6fde7791695\") " pod="openshift-network-operator/iptables-alerter-gqftj" Jul 17 17:56:47.195777 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195657 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-host\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.195777 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195684 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-run\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.195777 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195712 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a78443b7-6add-4ea0-9b24-949b0b9d117a-ovnkube-script-lib\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.195777 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195743 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltd2r\" (UniqueName: \"kubernetes.io/projected/a78443b7-6add-4ea0-9b24-949b0b9d117a-kube-api-access-ltd2r\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.195777 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195761 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-host\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.196434 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195769 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-node-log\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.196434 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195858 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48aa1cde-e776-496a-9816-9a67614f0210-konnectivity-ca\") pod \"konnectivity-agent-xbw8z\" (UID: \"48aa1cde-e776-496a-9816-9a67614f0210\") " pod="kube-system/konnectivity-agent-xbw8z" Jul 17 17:56:47.196434 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195890 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs\") pod \"network-metrics-daemon-5ccsk\" (UID: \"e05c7832-1be4-46bd-91e4-2dd5c8abeb02\") " pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:56:47.196434 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195916 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19dad044-c54d-4bad-adf5-dcf705460cac-cnibin\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.196434 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195860 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-run\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.196434 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195820 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-node-log\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.196434 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195946 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19dad044-c54d-4bad-adf5-dcf705460cac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.196434 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.195989 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-etc-modprobe-d\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.196434 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196035 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-run-openvswitch\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.196434 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196062 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0bc2b05e-ac19-449a-b861-d729822c0afa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.196434 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196087 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0bc2b05e-ac19-449a-b861-d729822c0afa-etc-selinux\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.196434 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196115 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqpgh\" (UniqueName: \"kubernetes.io/projected/e4c7e486-1512-437e-889e-678ce65751c7-kube-api-access-jqpgh\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.196434 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196126 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-run-openvswitch\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.196434 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196138 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-etc-modprobe-d\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.196434 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196146 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-etc-sysconfig\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.196434 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196160 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0bc2b05e-ac19-449a-b861-d729822c0afa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.196434 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196199 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-etc-sysconfig\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.197246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196211 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-host-run-netns\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.197246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196242 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-host-var-lib-cni-bin\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.197246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196247 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0bc2b05e-ac19-449a-b861-d729822c0afa-etc-selinux\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.197246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196270 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-multus-conf-dir\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.197246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196299 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-lib-modules\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.197246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196326 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/19dad044-c54d-4bad-adf5-dcf705460cac-os-release\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.197246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196355 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-var-lib-openvswitch\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.197246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196381 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-cni-netd\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.197246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196410 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19dad044-c54d-4bad-adf5-dcf705460cac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.197246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196415 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-var-lib-openvswitch\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.197246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196429 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-cni-netd\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.197246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196438 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-multus-socket-dir-parent\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.197246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196448 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-lib-modules\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.197246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196466 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fefc9465-ec24-48f7-9a99-d6fde7791695-iptables-alerter-script\") pod \"iptables-alerter-gqftj\" (UID: \"fefc9465-ec24-48f7-9a99-d6fde7791695\") " pod="openshift-network-operator/iptables-alerter-gqftj" Jul 17 17:56:47.197246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196493 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fefc9465-ec24-48f7-9a99-d6fde7791695-host-slash\") pod \"iptables-alerter-gqftj\" (UID: \"fefc9465-ec24-48f7-9a99-d6fde7791695\") " pod="openshift-network-operator/iptables-alerter-gqftj" Jul 17 17:56:47.197246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196501 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48aa1cde-e776-496a-9816-9a67614f0210-konnectivity-ca\") pod \"konnectivity-agent-xbw8z\" (UID: \"48aa1cde-e776-496a-9816-9a67614f0210\") " pod="kube-system/konnectivity-agent-xbw8z" Jul 17 17:56:47.197246 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196517 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/118b7937-fc4d-48fa-927e-2368a2317850-kubelet-config\") pod \"global-pull-secret-syncer-ts454\" (UID: \"118b7937-fc4d-48fa-927e-2368a2317850\") " pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:47.198184 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196530 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a78443b7-6add-4ea0-9b24-949b0b9d117a-ovnkube-script-lib\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.198184 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196555 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rlsn\" (UniqueName: \"kubernetes.io/projected/2e007a42-9178-41c7-bd04-0544e3996bce-kube-api-access-5rlsn\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.198184 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196585 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-kubelet\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.198184 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196627 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-kubelet\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.198184 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196682 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-run-systemd\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.198184 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196712 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-sys\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.198184 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196745 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e007a42-9178-41c7-bd04-0544e3996bce-tmp\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.198184 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196812 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-sys\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.198184 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196806 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-run-systemd\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.198184 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196830 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0bc2b05e-ac19-449a-b861-d729822c0afa-socket-dir\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.198184 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196864 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0bc2b05e-ac19-449a-b861-d729822c0afa-registration-dir\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.198184 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196890 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm5wk\" (UniqueName: \"kubernetes.io/projected/0bc2b05e-ac19-449a-b861-d729822c0afa-kube-api-access-lm5wk\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.198184 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196906 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0bc2b05e-ac19-449a-b861-d729822c0afa-socket-dir\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.198184 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196916 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-etc-sysctl-d\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.198184 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196939 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0bc2b05e-ac19-449a-b861-d729822c0afa-registration-dir\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.198184 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196941 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e007a42-9178-41c7-bd04-0544e3996bce-etc-tuned\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.198184 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.196966 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-log-socket\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.199012 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197024 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a78443b7-6add-4ea0-9b24-949b0b9d117a-ovn-node-metrics-cert\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.199012 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197056 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0bc2b05e-ac19-449a-b861-d729822c0afa-sys-fs\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.199012 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197075 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-system-cni-dir\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.199012 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197093 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-os-release\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.199012 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197116 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-run-netns\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.199012 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197143 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-run-ovn-kubernetes\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.199012 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197134 2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 17 17:56:47.199012 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197171 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.199012 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197204 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bkk8\" (UniqueName: \"kubernetes.io/projected/19dad044-c54d-4bad-adf5-dcf705460cac-kube-api-access-6bkk8\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.199012 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197220 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-cnibin\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.199012 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197239 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-host-var-lib-cni-multus\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.199012 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197269 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-host-var-lib-kubelet\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.199012 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197303 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-hostroot\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.199012 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197325 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48aa1cde-e776-496a-9816-9a67614f0210-agent-certs\") pod \"konnectivity-agent-xbw8z\" (UID: \"48aa1cde-e776-496a-9816-9a67614f0210\") " pod="kube-system/konnectivity-agent-xbw8z" Jul 17 17:56:47.199012 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197360 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-cni-bin\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.199012 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197387 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltg2b\" (UniqueName: \"kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b\") pod \"network-check-target-wcf2q\" (UID: \"27188dfc-708b-462f-9a9a-24fda6997009\") " pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:56:47.199012 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197414 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4c7e486-1512-437e-889e-678ce65751c7-cni-binary-copy\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.199942 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197440 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e4c7e486-1512-437e-889e-678ce65751c7-multus-daemon-config\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.199942 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197478 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/118b7937-fc4d-48fa-927e-2368a2317850-dbus\") pod \"global-pull-secret-syncer-ts454\" (UID: \"118b7937-fc4d-48fa-927e-2368a2317850\") " pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:47.199942 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197505 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-slash\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.199942 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197515 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-log-socket\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.199942 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197532 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a78443b7-6add-4ea0-9b24-949b0b9d117a-ovnkube-config\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.199942 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197558 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c8841f5d-e09f-4e42-b700-0002b5a3a342-serviceca\") pod \"node-ca-hfrgj\" (UID: \"c8841f5d-e09f-4e42-b700-0002b5a3a342\") " pod="openshift-image-registry/node-ca-hfrgj" Jul 17 17:56:47.199942 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197589 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6xln\" (UniqueName: \"kubernetes.io/projected/c8841f5d-e09f-4e42-b700-0002b5a3a342-kube-api-access-r6xln\") pod \"node-ca-hfrgj\" (UID: \"c8841f5d-e09f-4e42-b700-0002b5a3a342\") " pod="openshift-image-registry/node-ca-hfrgj" Jul 17 17:56:47.199942 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197600 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0bc2b05e-ac19-449a-b861-d729822c0afa-sys-fs\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.199942 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197615 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95sst\" (UniqueName: \"kubernetes.io/projected/376644cc-94a0-41d2-9cc0-cc410806ecf8-kube-api-access-95sst\") pod \"node-resolver-gpnwp\" (UID: \"376644cc-94a0-41d2-9cc0-cc410806ecf8\") " pod="openshift-dns/node-resolver-gpnwp" Jul 17 17:56:47.199942 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197643 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19dad044-c54d-4bad-adf5-dcf705460cac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.199942 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197880 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-run-netns\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.199942 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197931 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-run-ovn-kubernetes\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.199942 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.197970 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.199942 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198006 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-host-run-multus-certs\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.199942 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198032 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret\") pod \"global-pull-secret-syncer-ts454\" (UID: \"118b7937-fc4d-48fa-927e-2368a2317850\") " pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:47.199942 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198063 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-etc-kubernetes\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.199942 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198103 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-etc-openvswitch\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.200884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198132 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0bc2b05e-ac19-449a-b861-d729822c0afa-device-dir\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.200884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198158 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-systemd-units\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.200884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198183 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a78443b7-6add-4ea0-9b24-949b0b9d117a-env-overrides\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.200884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198211 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/376644cc-94a0-41d2-9cc0-cc410806ecf8-tmp-dir\") pod \"node-resolver-gpnwp\" (UID: \"376644cc-94a0-41d2-9cc0-cc410806ecf8\") " pod="openshift-dns/node-resolver-gpnwp" Jul 17 17:56:47.200884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198263 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-etc-sysctl-d\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.200884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198311 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19dad044-c54d-4bad-adf5-dcf705460cac-cni-binary-copy\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.200884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198342 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-host-run-k8s-cni-cncf-io\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.200884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198373 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-etc-sysctl-conf\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.200884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198402 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-run-ovn\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.200884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198428 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19dad044-c54d-4bad-adf5-dcf705460cac-system-cni-dir\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.200884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198752 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/376644cc-94a0-41d2-9cc0-cc410806ecf8-tmp-dir\") pod \"node-resolver-gpnwp\" (UID: \"376644cc-94a0-41d2-9cc0-cc410806ecf8\") " pod="openshift-dns/node-resolver-gpnwp" Jul 17 17:56:47.200884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198820 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a78443b7-6add-4ea0-9b24-949b0b9d117a-ovnkube-config\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.200884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198835 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-slash\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.200884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198914 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-host-cni-bin\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.200884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.198967 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-etc-sysctl-conf\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.200884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.199343 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c8841f5d-e09f-4e42-b700-0002b5a3a342-serviceca\") pod \"node-ca-hfrgj\" (UID: \"c8841f5d-e09f-4e42-b700-0002b5a3a342\") " pod="openshift-image-registry/node-ca-hfrgj" Jul 17 17:56:47.200884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.199399 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-systemd-units\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.201747 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.199421 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0bc2b05e-ac19-449a-b861-d729822c0afa-device-dir\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.201747 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.199469 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-run-ovn\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.201747 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.199478 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a78443b7-6add-4ea0-9b24-949b0b9d117a-etc-openvswitch\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.201747 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.199533 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e007a42-9178-41c7-bd04-0544e3996bce-etc-kubernetes\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.201747 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.199773 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a78443b7-6add-4ea0-9b24-949b0b9d117a-env-overrides\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.202023 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.201998 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e007a42-9178-41c7-bd04-0544e3996bce-tmp\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.202419 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.202394 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a78443b7-6add-4ea0-9b24-949b0b9d117a-ovn-node-metrics-cert\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.204273 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.204249 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e007a42-9178-41c7-bd04-0544e3996bce-etc-tuned\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.204387 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.204270 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48aa1cde-e776-496a-9816-9a67614f0210-agent-certs\") pod \"konnectivity-agent-xbw8z\" (UID: \"48aa1cde-e776-496a-9816-9a67614f0210\") " pod="kube-system/konnectivity-agent-xbw8z" Jul 17 17:56:47.207604 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.206836 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:56:47.207604 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.206862 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:56:47.207604 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.206876 2572 projected.go:196] Error preparing data for projected volume kube-api-access-ltg2b for pod openshift-network-diagnostics/network-check-target-wcf2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:56:47.207604 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.206960 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b podName:27188dfc-708b-462f-9a9a-24fda6997009 nodeName:}" failed. No retries permitted until 2026-07-17 17:56:47.706930295 +0000 UTC m=+3.116357888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ltg2b" (UniqueName: "kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b") pod "network-check-target-wcf2q" (UID: "27188dfc-708b-462f-9a9a-24fda6997009") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:56:47.209379 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.209348 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltd2r\" (UniqueName: \"kubernetes.io/projected/a78443b7-6add-4ea0-9b24-949b0b9d117a-kube-api-access-ltd2r\") pod \"ovnkube-node-8v7vg\" (UID: \"a78443b7-6add-4ea0-9b24-949b0b9d117a\") " pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.209490 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.209348 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rlsn\" (UniqueName: \"kubernetes.io/projected/2e007a42-9178-41c7-bd04-0544e3996bce-kube-api-access-5rlsn\") pod \"tuned-nwszv\" (UID: \"2e007a42-9178-41c7-bd04-0544e3996bce\") " pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.209783 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.209759 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6xln\" (UniqueName: \"kubernetes.io/projected/c8841f5d-e09f-4e42-b700-0002b5a3a342-kube-api-access-r6xln\") pod \"node-ca-hfrgj\" (UID: \"c8841f5d-e09f-4e42-b700-0002b5a3a342\") " pod="openshift-image-registry/node-ca-hfrgj" Jul 17 17:56:47.210369 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.210345 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm5wk\" (UniqueName: \"kubernetes.io/projected/0bc2b05e-ac19-449a-b861-d729822c0afa-kube-api-access-lm5wk\") pod \"aws-ebs-csi-driver-node-hh5r2\" (UID: \"0bc2b05e-ac19-449a-b861-d729822c0afa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.211476 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.211455 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95sst\" (UniqueName: \"kubernetes.io/projected/376644cc-94a0-41d2-9cc0-cc410806ecf8-kube-api-access-95sst\") pod \"node-resolver-gpnwp\" (UID: \"376644cc-94a0-41d2-9cc0-cc410806ecf8\") " pod="openshift-dns/node-resolver-gpnwp" Jul 17 17:56:47.224130 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.224050 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-62.ec2.internal" event={"ID":"8b28b739fa6bf195253cfa6375ca90e6","Type":"ContainerStarted","Data":"618d8f17b14067e0ded1091dbb7c7ef88b8bb8cec531c052de3d93505e508b3a"} Jul 17 17:56:47.225560 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.225533 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal" event={"ID":"286e9012d20d5d00cdc3b7cfc23fe73f","Type":"ContainerStarted","Data":"db91719345149e7cfd071c3b26c5fc1040924dc3c1a53371e9f12f2f63cc206d"} Jul 17 17:56:47.298789 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.298751 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-system-cni-dir\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.298789 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.298797 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-os-release\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299089 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.298823 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bkk8\" (UniqueName: \"kubernetes.io/projected/19dad044-c54d-4bad-adf5-dcf705460cac-kube-api-access-6bkk8\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.299089 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.298849 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-cnibin\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299089 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.298871 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-host-var-lib-cni-multus\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299089 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.298895 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-host-var-lib-kubelet\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299089 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.298910 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-system-cni-dir\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299089 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.298927 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-os-release\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299089 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.298971 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-hostroot\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299089 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.298972 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-cnibin\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299089 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.298981 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-host-var-lib-cni-multus\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299089 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.298919 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-hostroot\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299089 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299007 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-host-var-lib-kubelet\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299089 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299053 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4c7e486-1512-437e-889e-678ce65751c7-cni-binary-copy\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299089 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299072 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e4c7e486-1512-437e-889e-678ce65751c7-multus-daemon-config\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299089 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299088 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/118b7937-fc4d-48fa-927e-2368a2317850-dbus\") pod \"global-pull-secret-syncer-ts454\" (UID: \"118b7937-fc4d-48fa-927e-2368a2317850\") " pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:47.299785 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299110 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19dad044-c54d-4bad-adf5-dcf705460cac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.299785 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299135 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-host-run-multus-certs\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299785 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299190 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret\") pod \"global-pull-secret-syncer-ts454\" (UID: \"118b7937-fc4d-48fa-927e-2368a2317850\") " pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:47.299785 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299219 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19dad044-c54d-4bad-adf5-dcf705460cac-cni-binary-copy\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.299785 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299243 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-host-run-k8s-cni-cncf-io\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299785 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299268 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19dad044-c54d-4bad-adf5-dcf705460cac-system-cni-dir\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.299785 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299271 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-host-run-multus-certs\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299785 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299295 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n4qp\" (UniqueName: \"kubernetes.io/projected/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-kube-api-access-9n4qp\") pod \"network-metrics-daemon-5ccsk\" (UID: \"e05c7832-1be4-46bd-91e4-2dd5c8abeb02\") " pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:56:47.299785 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299302 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19dad044-c54d-4bad-adf5-dcf705460cac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.299785 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.299317 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:56:47.299785 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299327 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-host-run-k8s-cni-cncf-io\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299785 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299328 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-multus-cni-dir\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299785 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299349 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19dad044-c54d-4bad-adf5-dcf705460cac-system-cni-dir\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.299785 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299365 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-etc-kubernetes\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299785 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.299389 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret podName:118b7937-fc4d-48fa-927e-2368a2317850 nodeName:}" failed. No retries permitted until 2026-07-17 17:56:47.799368525 +0000 UTC m=+3.208796131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret") pod "global-pull-secret-syncer-ts454" (UID: "118b7937-fc4d-48fa-927e-2368a2317850") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:56:47.299785 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299399 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-etc-kubernetes\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.299785 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299426 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkbwn\" (UniqueName: \"kubernetes.io/projected/fefc9465-ec24-48f7-9a99-d6fde7791695-kube-api-access-bkbwn\") pod \"iptables-alerter-gqftj\" (UID: \"fefc9465-ec24-48f7-9a99-d6fde7791695\") " pod="openshift-network-operator/iptables-alerter-gqftj" Jul 17 17:56:47.300680 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299458 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs\") pod \"network-metrics-daemon-5ccsk\" (UID: \"e05c7832-1be4-46bd-91e4-2dd5c8abeb02\") " pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:56:47.300680 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299483 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19dad044-c54d-4bad-adf5-dcf705460cac-cnibin\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.300680 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299508 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19dad044-c54d-4bad-adf5-dcf705460cac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.300680 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299539 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqpgh\" (UniqueName: \"kubernetes.io/projected/e4c7e486-1512-437e-889e-678ce65751c7-kube-api-access-jqpgh\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.300680 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299565 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-host-run-netns\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.300680 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299590 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-host-var-lib-cni-bin\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.300680 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299614 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-multus-conf-dir\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.300680 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299640 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/19dad044-c54d-4bad-adf5-dcf705460cac-os-release\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.300680 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299666 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19dad044-c54d-4bad-adf5-dcf705460cac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.300680 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299695 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-multus-socket-dir-parent\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.300680 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299722 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fefc9465-ec24-48f7-9a99-d6fde7791695-iptables-alerter-script\") pod \"iptables-alerter-gqftj\" (UID: \"fefc9465-ec24-48f7-9a99-d6fde7791695\") " pod="openshift-network-operator/iptables-alerter-gqftj" Jul 17 17:56:47.300680 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.299747 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:56:47.300680 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299785 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fefc9465-ec24-48f7-9a99-d6fde7791695-host-slash\") pod \"iptables-alerter-gqftj\" (UID: \"fefc9465-ec24-48f7-9a99-d6fde7791695\") " pod="openshift-network-operator/iptables-alerter-gqftj" Jul 17 17:56:47.300680 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.299794 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs podName:e05c7832-1be4-46bd-91e4-2dd5c8abeb02 nodeName:}" failed. No retries permitted until 2026-07-17 17:56:47.799776747 +0000 UTC m=+3.209204349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs") pod "network-metrics-daemon-5ccsk" (UID: "e05c7832-1be4-46bd-91e4-2dd5c8abeb02") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:56:47.300680 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299794 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4c7e486-1512-437e-889e-678ce65751c7-cni-binary-copy\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.300680 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299748 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fefc9465-ec24-48f7-9a99-d6fde7791695-host-slash\") pod \"iptables-alerter-gqftj\" (UID: \"fefc9465-ec24-48f7-9a99-d6fde7791695\") " pod="openshift-network-operator/iptables-alerter-gqftj" Jul 17 17:56:47.300680 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299834 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-host-run-netns\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.301627 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299846 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e4c7e486-1512-437e-889e-678ce65751c7-multus-daemon-config\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.301627 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299855 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/118b7937-fc4d-48fa-927e-2368a2317850-kubelet-config\") pod \"global-pull-secret-syncer-ts454\" (UID: \"118b7937-fc4d-48fa-927e-2368a2317850\") " pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:47.301627 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299894 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/19dad044-c54d-4bad-adf5-dcf705460cac-os-release\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.301627 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299739 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19dad044-c54d-4bad-adf5-dcf705460cac-cnibin\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.301627 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299899 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-host-var-lib-cni-bin\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.301627 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299951 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/118b7937-fc4d-48fa-927e-2368a2317850-kubelet-config\") pod \"global-pull-secret-syncer-ts454\" (UID: \"118b7937-fc4d-48fa-927e-2368a2317850\") " pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:47.301627 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299970 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-multus-conf-dir\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.301627 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.299973 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19dad044-c54d-4bad-adf5-dcf705460cac-cni-binary-copy\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.301627 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.300003 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-multus-socket-dir-parent\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.301627 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.300049 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/118b7937-fc4d-48fa-927e-2368a2317850-dbus\") pod \"global-pull-secret-syncer-ts454\" (UID: \"118b7937-fc4d-48fa-927e-2368a2317850\") " pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:47.301627 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.300115 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4c7e486-1512-437e-889e-678ce65751c7-multus-cni-dir\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.301627 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.300118 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19dad044-c54d-4bad-adf5-dcf705460cac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.301627 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.300348 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19dad044-c54d-4bad-adf5-dcf705460cac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.301627 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.300463 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fefc9465-ec24-48f7-9a99-d6fde7791695-iptables-alerter-script\") pod \"iptables-alerter-gqftj\" (UID: \"fefc9465-ec24-48f7-9a99-d6fde7791695\") " pod="openshift-network-operator/iptables-alerter-gqftj" Jul 17 17:56:47.310166 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.310132 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bkk8\" (UniqueName: \"kubernetes.io/projected/19dad044-c54d-4bad-adf5-dcf705460cac-kube-api-access-6bkk8\") pod \"multus-additional-cni-plugins-sz4tq\" (UID: \"19dad044-c54d-4bad-adf5-dcf705460cac\") " pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.310934 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.310864 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n4qp\" (UniqueName: \"kubernetes.io/projected/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-kube-api-access-9n4qp\") pod \"network-metrics-daemon-5ccsk\" (UID: \"e05c7832-1be4-46bd-91e4-2dd5c8abeb02\") " pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:56:47.311231 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.311209 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqpgh\" (UniqueName: \"kubernetes.io/projected/e4c7e486-1512-437e-889e-678ce65751c7-kube-api-access-jqpgh\") pod \"multus-m8chw\" (UID: \"e4c7e486-1512-437e-889e-678ce65751c7\") " pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.311355 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.311260 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkbwn\" (UniqueName: \"kubernetes.io/projected/fefc9465-ec24-48f7-9a99-d6fde7791695-kube-api-access-bkbwn\") pod \"iptables-alerter-gqftj\" (UID: \"fefc9465-ec24-48f7-9a99-d6fde7791695\") " pod="openshift-network-operator/iptables-alerter-gqftj" Jul 17 17:56:47.378422 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.378385 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nwszv" Jul 17 17:56:47.388987 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.388962 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" Jul 17 17:56:47.398040 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.397999 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gpnwp" Jul 17 17:56:47.411028 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.411005 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:56:47.420069 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.420046 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hfrgj" Jul 17 17:56:47.427913 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.427888 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xbw8z" Jul 17 17:56:47.435833 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.435813 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sz4tq" Jul 17 17:56:47.443799 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.443777 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m8chw" Jul 17 17:56:47.451363 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.451342 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gqftj" Jul 17 17:56:47.581893 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.581816 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:56:47.802833 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.802791 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs\") pod \"network-metrics-daemon-5ccsk\" (UID: \"e05c7832-1be4-46bd-91e4-2dd5c8abeb02\") " pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:56:47.803018 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.802863 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltg2b\" (UniqueName: \"kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b\") pod \"network-check-target-wcf2q\" (UID: \"27188dfc-708b-462f-9a9a-24fda6997009\") " pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:56:47.803018 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:47.802902 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret\") pod \"global-pull-secret-syncer-ts454\" (UID: \"118b7937-fc4d-48fa-927e-2368a2317850\") " pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:47.803018 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.802981 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:56:47.803233 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.803073 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:56:47.803233 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.803088 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:56:47.803233 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.803101 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs podName:e05c7832-1be4-46bd-91e4-2dd5c8abeb02 nodeName:}" failed. No retries permitted until 2026-07-17 17:56:48.803060359 +0000 UTC m=+4.212487960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs") pod "network-metrics-daemon-5ccsk" (UID: "e05c7832-1be4-46bd-91e4-2dd5c8abeb02") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:56:47.803233 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.803113 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:56:47.803233 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.803136 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret podName:118b7937-fc4d-48fa-927e-2368a2317850 nodeName:}" failed. No retries permitted until 2026-07-17 17:56:48.803124272 +0000 UTC m=+4.212551861 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret") pod "global-pull-secret-syncer-ts454" (UID: "118b7937-fc4d-48fa-927e-2368a2317850") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:56:47.803233 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.803139 2572 projected.go:196] Error preparing data for projected volume kube-api-access-ltg2b for pod openshift-network-diagnostics/network-check-target-wcf2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:56:47.803233 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:47.803191 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b podName:27188dfc-708b-462f-9a9a-24fda6997009 nodeName:}" failed. No retries permitted until 2026-07-17 17:56:48.803174417 +0000 UTC m=+4.212602007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ltg2b" (UniqueName: "kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b") pod "network-check-target-wcf2q" (UID: "27188dfc-708b-462f-9a9a-24fda6997009") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:56:48.136199 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:48.136123 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 17:51:47 +0000 UTC" deadline="2028-01-11 14:40:15.815103622 +0000 UTC" Jul 17 17:56:48.136199 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:48.136194 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13028h43m27.678914424s" Jul 17 17:56:48.220047 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:48.219971 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:48.220285 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:48.220166 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:56:48.284208 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:48.284175 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48aa1cde_e776_496a_9816_9a67614f0210.slice/crio-a66b898a23c9d37b4d4019d3370b37c813ea6d44f5d0fc5aa328afea0695c739 WatchSource:0}: Error finding container a66b898a23c9d37b4d4019d3370b37c813ea6d44f5d0fc5aa328afea0695c739: Status 404 returned error can't find the container with id a66b898a23c9d37b4d4019d3370b37c813ea6d44f5d0fc5aa328afea0695c739 Jul 17 17:56:48.286827 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:48.286798 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19dad044_c54d_4bad_adf5_dcf705460cac.slice/crio-21249945d64bf1ae36458df81ae0f15dc83720313d2e1496b769cb2231d0b270 WatchSource:0}: Error finding container 21249945d64bf1ae36458df81ae0f15dc83720313d2e1496b769cb2231d0b270: Status 404 returned error can't find the container with id 21249945d64bf1ae36458df81ae0f15dc83720313d2e1496b769cb2231d0b270 Jul 17 17:56:48.289980 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:48.289951 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc2b05e_ac19_449a_b861_d729822c0afa.slice/crio-48cc2fe0efa85f8279d905ecf521b9d6ecd26360f0b427db34a6c197f7c6a224 WatchSource:0}: Error finding container 48cc2fe0efa85f8279d905ecf521b9d6ecd26360f0b427db34a6c197f7c6a224: Status 404 returned error can't find the container with id 48cc2fe0efa85f8279d905ecf521b9d6ecd26360f0b427db34a6c197f7c6a224 Jul 17 17:56:48.291172 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:48.291146 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78443b7_6add_4ea0_9b24_949b0b9d117a.slice/crio-3d71c11f1abac7e58f9a3f951753688f95a36ece37847fd2a32c5dd81f941680 WatchSource:0}: Error finding container 3d71c11f1abac7e58f9a3f951753688f95a36ece37847fd2a32c5dd81f941680: Status 404 returned error can't find the container with id 3d71c11f1abac7e58f9a3f951753688f95a36ece37847fd2a32c5dd81f941680 Jul 17 17:56:48.292390 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:48.292284 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefc9465_ec24_48f7_9a99_d6fde7791695.slice/crio-c06046c1660d05758d9feba8444a43058d0d632e2631628d8b91f21e0e3a2527 WatchSource:0}: Error finding container c06046c1660d05758d9feba8444a43058d0d632e2631628d8b91f21e0e3a2527: Status 404 returned error can't find the container with id c06046c1660d05758d9feba8444a43058d0d632e2631628d8b91f21e0e3a2527 Jul 17 17:56:48.293339 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:48.293307 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c7e486_1512_437e_889e_678ce65751c7.slice/crio-35b684c909aaab924471a869af3814adf0189c5f14c6296481e44c0e3c4d1ba0 WatchSource:0}: Error finding container 35b684c909aaab924471a869af3814adf0189c5f14c6296481e44c0e3c4d1ba0: Status 404 returned error can't find the container with id 35b684c909aaab924471a869af3814adf0189c5f14c6296481e44c0e3c4d1ba0 Jul 17 17:56:48.295780 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:56:48.294908 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8841f5d_e09f_4e42_b700_0002b5a3a342.slice/crio-5a30bb62b318eaca36325b563ba415628c5b914011449f79ead568210b054e29 WatchSource:0}: Error finding container 5a30bb62b318eaca36325b563ba415628c5b914011449f79ead568210b054e29: Status 404 returned error can't find the container with id 5a30bb62b318eaca36325b563ba415628c5b914011449f79ead568210b054e29 Jul 17 17:56:48.811639 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:48.811319 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltg2b\" (UniqueName: \"kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b\") pod \"network-check-target-wcf2q\" (UID: \"27188dfc-708b-462f-9a9a-24fda6997009\") " pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:56:48.811639 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:48.811629 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret\") pod \"global-pull-secret-syncer-ts454\" (UID: \"118b7937-fc4d-48fa-927e-2368a2317850\") " pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:48.811866 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:48.811684 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs\") pod \"network-metrics-daemon-5ccsk\" (UID: \"e05c7832-1be4-46bd-91e4-2dd5c8abeb02\") " pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:56:48.811866 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:48.811843 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:56:48.811968 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:48.811911 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs podName:e05c7832-1be4-46bd-91e4-2dd5c8abeb02 nodeName:}" failed. No retries permitted until 2026-07-17 17:56:50.811889509 +0000 UTC m=+6.221317117 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs") pod "network-metrics-daemon-5ccsk" (UID: "e05c7832-1be4-46bd-91e4-2dd5c8abeb02") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:56:48.812404 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:48.812381 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:56:48.812494 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:48.812411 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:56:48.812494 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:48.812426 2572 projected.go:196] Error preparing data for projected volume kube-api-access-ltg2b for pod openshift-network-diagnostics/network-check-target-wcf2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:56:48.812494 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:48.812479 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b podName:27188dfc-708b-462f-9a9a-24fda6997009 nodeName:}" failed. No retries permitted until 2026-07-17 17:56:50.81246247 +0000 UTC m=+6.221890062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ltg2b" (UniqueName: "kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b") pod "network-check-target-wcf2q" (UID: "27188dfc-708b-462f-9a9a-24fda6997009") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:56:48.812663 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:48.812547 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:56:48.812663 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:48.812581 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret podName:118b7937-fc4d-48fa-927e-2368a2317850 nodeName:}" failed. No retries permitted until 2026-07-17 17:56:50.812569092 +0000 UTC m=+6.221996678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret") pod "global-pull-secret-syncer-ts454" (UID: "118b7937-fc4d-48fa-927e-2368a2317850") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:56:49.136884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:49.136770 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 17:51:47 +0000 UTC" deadline="2028-02-26 14:01:04.102544533 +0000 UTC" Jul 17 17:56:49.136884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:49.136817 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14132h4m14.965732435s" Jul 17 17:56:49.226276 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:49.226243 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:56:49.226439 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:49.226400 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:56:49.226519 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:49.226456 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:56:49.226673 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:49.226648 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:56:49.245469 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:49.245429 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xbw8z" event={"ID":"48aa1cde-e776-496a-9816-9a67614f0210","Type":"ContainerStarted","Data":"a66b898a23c9d37b4d4019d3370b37c813ea6d44f5d0fc5aa328afea0695c739"} Jul 17 17:56:49.250918 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:49.250885 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-62.ec2.internal" event={"ID":"8b28b739fa6bf195253cfa6375ca90e6","Type":"ContainerStarted","Data":"6de1682b77d01dde2382a8c40ff068655a67b794b006b741b401a4304333d626"} Jul 17 17:56:49.259741 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:49.259714 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nwszv" event={"ID":"2e007a42-9178-41c7-bd04-0544e3996bce","Type":"ContainerStarted","Data":"d7312b93e68b66719f646fb4fec496b8d98941026e9ae6c9aa2030277e58f5ba"} Jul 17 17:56:49.271051 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:49.271020 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hfrgj" event={"ID":"c8841f5d-e09f-4e42-b700-0002b5a3a342","Type":"ContainerStarted","Data":"5a30bb62b318eaca36325b563ba415628c5b914011449f79ead568210b054e29"} Jul 17 17:56:49.277981 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:49.277908 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m8chw" event={"ID":"e4c7e486-1512-437e-889e-678ce65751c7","Type":"ContainerStarted","Data":"35b684c909aaab924471a869af3814adf0189c5f14c6296481e44c0e3c4d1ba0"} Jul 17 17:56:49.288240 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:49.288184 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gqftj" event={"ID":"fefc9465-ec24-48f7-9a99-d6fde7791695","Type":"ContainerStarted","Data":"c06046c1660d05758d9feba8444a43058d0d632e2631628d8b91f21e0e3a2527"} Jul 17 17:56:49.299899 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:49.299869 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" event={"ID":"0bc2b05e-ac19-449a-b861-d729822c0afa","Type":"ContainerStarted","Data":"48cc2fe0efa85f8279d905ecf521b9d6ecd26360f0b427db34a6c197f7c6a224"} Jul 17 17:56:49.314484 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:49.314449 2572 generic.go:356] "Generic (PLEG): container finished" podID="286e9012d20d5d00cdc3b7cfc23fe73f" containerID="de8d8b3a1a878c9cfc1d1855cc49c65eeddefddf3b550b4831e416d5dbde19e9" exitCode=0 Jul 17 17:56:49.314637 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:49.314543 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal" event={"ID":"286e9012d20d5d00cdc3b7cfc23fe73f","Type":"ContainerDied","Data":"de8d8b3a1a878c9cfc1d1855cc49c65eeddefddf3b550b4831e416d5dbde19e9"} Jul 17 17:56:49.327579 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:49.327549 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gpnwp" event={"ID":"376644cc-94a0-41d2-9cc0-cc410806ecf8","Type":"ContainerStarted","Data":"f99ecaa4f1c51fda8632fe7c326930cfc21895d82dc368cc89de452a49d9d8ee"} Jul 17 17:56:49.330580 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:49.330552 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" event={"ID":"a78443b7-6add-4ea0-9b24-949b0b9d117a","Type":"ContainerStarted","Data":"3d71c11f1abac7e58f9a3f951753688f95a36ece37847fd2a32c5dd81f941680"} Jul 17 17:56:49.337007 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:49.336950 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-62.ec2.internal" podStartSLOduration=3.336930291 podStartE2EDuration="3.336930291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:56:49.266402393 +0000 UTC m=+4.675830000" watchObservedRunningTime="2026-07-17 17:56:49.336930291 +0000 UTC m=+4.746357902" Jul 17 17:56:49.345501 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:49.345445 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sz4tq" event={"ID":"19dad044-c54d-4bad-adf5-dcf705460cac","Type":"ContainerStarted","Data":"21249945d64bf1ae36458df81ae0f15dc83720313d2e1496b769cb2231d0b270"} Jul 17 17:56:50.220007 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:50.219976 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:50.220592 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:50.220133 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:56:50.358948 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:50.358819 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal" event={"ID":"286e9012d20d5d00cdc3b7cfc23fe73f","Type":"ContainerStarted","Data":"9bcbb74f1642ac67c8282bb23f57fc30459760203b11aa44394e73b69858889f"} Jul 17 17:56:50.836300 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:50.836258 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltg2b\" (UniqueName: \"kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b\") pod \"network-check-target-wcf2q\" (UID: \"27188dfc-708b-462f-9a9a-24fda6997009\") " pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:56:50.836503 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:50.836328 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret\") pod \"global-pull-secret-syncer-ts454\" (UID: \"118b7937-fc4d-48fa-927e-2368a2317850\") " pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:50.836503 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:50.836375 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs\") pod \"network-metrics-daemon-5ccsk\" (UID: \"e05c7832-1be4-46bd-91e4-2dd5c8abeb02\") " pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:56:50.836503 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:50.836477 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:56:50.836503 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:50.836499 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:56:50.836503 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:50.836505 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:56:50.836767 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:50.836522 2572 projected.go:196] Error preparing data for projected volume kube-api-access-ltg2b for pod openshift-network-diagnostics/network-check-target-wcf2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:56:50.836767 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:50.836562 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs podName:e05c7832-1be4-46bd-91e4-2dd5c8abeb02 nodeName:}" failed. No retries permitted until 2026-07-17 17:56:54.836543265 +0000 UTC m=+10.245970856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs") pod "network-metrics-daemon-5ccsk" (UID: "e05c7832-1be4-46bd-91e4-2dd5c8abeb02") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:56:50.836767 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:50.836584 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b podName:27188dfc-708b-462f-9a9a-24fda6997009 nodeName:}" failed. No retries permitted until 2026-07-17 17:56:54.836571808 +0000 UTC m=+10.245999399 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ltg2b" (UniqueName: "kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b") pod "network-check-target-wcf2q" (UID: "27188dfc-708b-462f-9a9a-24fda6997009") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:56:50.836767 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:50.836622 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:56:50.836767 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:50.836670 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret podName:118b7937-fc4d-48fa-927e-2368a2317850 nodeName:}" failed. No retries permitted until 2026-07-17 17:56:54.836648055 +0000 UTC m=+10.246075645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret") pod "global-pull-secret-syncer-ts454" (UID: "118b7937-fc4d-48fa-927e-2368a2317850") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:56:51.221914 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:51.221837 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:56:51.222375 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:51.221963 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:56:51.222547 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:51.222529 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:56:51.222678 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:51.222654 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:56:52.219245 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:52.219214 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:52.219421 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:52.219355 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:56:53.220154 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:53.220071 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:56:53.220614 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:53.220226 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:56:53.220614 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:53.220082 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:56:53.220614 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:53.220337 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:56:54.219967 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:54.219435 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:54.219967 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:54.219572 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:56:54.872234 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:54.872193 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltg2b\" (UniqueName: \"kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b\") pod \"network-check-target-wcf2q\" (UID: \"27188dfc-708b-462f-9a9a-24fda6997009\") " pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:56:54.872234 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:54.872249 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret\") pod \"global-pull-secret-syncer-ts454\" (UID: \"118b7937-fc4d-48fa-927e-2368a2317850\") " pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:54.872775 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:54.872290 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs\") pod \"network-metrics-daemon-5ccsk\" (UID: \"e05c7832-1be4-46bd-91e4-2dd5c8abeb02\") " pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:56:54.872775 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:54.872414 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:56:54.872775 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:54.872462 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs podName:e05c7832-1be4-46bd-91e4-2dd5c8abeb02 nodeName:}" failed. No retries permitted until 2026-07-17 17:57:02.872446352 +0000 UTC m=+18.281873950 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs") pod "network-metrics-daemon-5ccsk" (UID: "e05c7832-1be4-46bd-91e4-2dd5c8abeb02") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:56:54.872973 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:54.872853 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:56:54.872973 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:54.872870 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:56:54.872973 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:54.872883 2572 projected.go:196] Error preparing data for projected volume kube-api-access-ltg2b for pod openshift-network-diagnostics/network-check-target-wcf2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:56:54.872973 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:54.872927 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b podName:27188dfc-708b-462f-9a9a-24fda6997009 nodeName:}" failed. No retries permitted until 2026-07-17 17:57:02.872912767 +0000 UTC m=+18.282340365 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ltg2b" (UniqueName: "kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b") pod "network-check-target-wcf2q" (UID: "27188dfc-708b-462f-9a9a-24fda6997009") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:56:54.872973 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:54.872935 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:56:54.873271 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:54.873026 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret podName:118b7937-fc4d-48fa-927e-2368a2317850 nodeName:}" failed. No retries permitted until 2026-07-17 17:57:02.872994025 +0000 UTC m=+18.282421624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret") pod "global-pull-secret-syncer-ts454" (UID: "118b7937-fc4d-48fa-927e-2368a2317850") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:56:55.221075 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:55.220917 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:56:55.221075 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:55.221052 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:56:55.221632 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:55.221484 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:56:55.221632 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:55.221594 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:56:56.219220 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:56.219187 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:56.219659 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:56.219310 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:56:57.219875 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:57.219836 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:56:57.220366 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:57.219837 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:56:57.220366 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:57.219965 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:56:57.220366 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:57.220070 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:56:58.219354 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:58.219319 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:56:58.219537 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:58.219450 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:56:59.219192 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:59.219111 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:56:59.219606 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:56:59.219111 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:56:59.219606 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:59.219254 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:56:59.219606 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:56:59.219292 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:57:00.218989 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:00.218959 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:00.219184 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:00.219074 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:57:01.219316 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:01.219282 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:01.219795 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:01.219411 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:57:01.219795 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:01.219467 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:01.219795 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:01.219593 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:57:02.219693 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:02.219655 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:02.220222 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:02.219781 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:57:02.938040 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:02.937945 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs\") pod \"network-metrics-daemon-5ccsk\" (UID: \"e05c7832-1be4-46bd-91e4-2dd5c8abeb02\") " pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:02.938040 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:02.938022 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltg2b\" (UniqueName: \"kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b\") pod \"network-check-target-wcf2q\" (UID: \"27188dfc-708b-462f-9a9a-24fda6997009\") " pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:02.938249 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:02.938134 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:57:02.938249 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:02.938153 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:57:02.938249 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:02.938137 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:57:02.938249 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:02.938171 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret\") pod \"global-pull-secret-syncer-ts454\" (UID: \"118b7937-fc4d-48fa-927e-2368a2317850\") " pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:02.938249 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:02.938235 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs podName:e05c7832-1be4-46bd-91e4-2dd5c8abeb02 nodeName:}" failed. No retries permitted until 2026-07-17 17:57:18.938210369 +0000 UTC m=+34.347637955 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs") pod "network-metrics-daemon-5ccsk" (UID: "e05c7832-1be4-46bd-91e4-2dd5c8abeb02") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:57:02.938249 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:02.938168 2572 projected.go:196] Error preparing data for projected volume kube-api-access-ltg2b for pod openshift-network-diagnostics/network-check-target-wcf2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:57:02.938249 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:02.938252 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:57:02.938595 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:02.938281 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b podName:27188dfc-708b-462f-9a9a-24fda6997009 nodeName:}" failed. No retries permitted until 2026-07-17 17:57:18.938271411 +0000 UTC m=+34.347699022 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ltg2b" (UniqueName: "kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b") pod "network-check-target-wcf2q" (UID: "27188dfc-708b-462f-9a9a-24fda6997009") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:57:02.938595 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:02.938297 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret podName:118b7937-fc4d-48fa-927e-2368a2317850 nodeName:}" failed. No retries permitted until 2026-07-17 17:57:18.938289596 +0000 UTC m=+34.347717180 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret") pod "global-pull-secret-syncer-ts454" (UID: "118b7937-fc4d-48fa-927e-2368a2317850") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:57:03.219781 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:03.219692 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:03.220239 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:03.219698 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:03.220239 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:03.219813 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:57:03.220239 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:03.219927 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:57:04.219369 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:04.219334 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:04.219536 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:04.219464 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:57:05.220174 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:05.220090 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:05.220597 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:05.220208 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:57:05.220597 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:05.220266 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:05.220597 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:05.220403 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:57:06.219672 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:06.219636 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:06.219847 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:06.219763 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:57:07.219523 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.219206 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:07.220349 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.219264 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:07.220349 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:07.219643 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:57:07.220349 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:07.219693 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:57:07.393796 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.393766 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hfrgj" event={"ID":"c8841f5d-e09f-4e42-b700-0002b5a3a342","Type":"ContainerStarted","Data":"d077e584c54b20c3e5ebe59b0779c781ff9b86ced05c7f491bb60ac2c4e230c4"} Jul 17 17:57:07.395122 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.395094 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m8chw" event={"ID":"e4c7e486-1512-437e-889e-678ce65751c7","Type":"ContainerStarted","Data":"fa6a7fc73402ce4b5c9d9e0b592fd4827a6a64f24bd3476cf68d932718beaf36"} Jul 17 17:57:07.396571 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.396548 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" event={"ID":"0bc2b05e-ac19-449a-b861-d729822c0afa","Type":"ContainerStarted","Data":"636385ac32ad5726891dc997e464ba66fd25a39f3df48e86265d8bb844b21d96"} Jul 17 17:57:07.397879 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.397858 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gpnwp" event={"ID":"376644cc-94a0-41d2-9cc0-cc410806ecf8","Type":"ContainerStarted","Data":"d62f7667e360f1633ec1ebe10a3c89c72107e7fac4635c4d48756678d8e21907"} Jul 17 17:57:07.401971 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.401950 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" event={"ID":"a78443b7-6add-4ea0-9b24-949b0b9d117a","Type":"ContainerStarted","Data":"c632ffa3cb365d96ccb35505a2f02dbd768cb75879257e5f303bcb47b4364ab8"} Jul 17 17:57:07.402077 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.402022 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" event={"ID":"a78443b7-6add-4ea0-9b24-949b0b9d117a","Type":"ContainerStarted","Data":"98947f4188ac3ee9e6b09d9e44e411dc820a11888aa5182ac3ddd39a4f645236"} Jul 17 17:57:07.402077 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.402059 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" event={"ID":"a78443b7-6add-4ea0-9b24-949b0b9d117a","Type":"ContainerStarted","Data":"73da3ed4c932c89fb0f03d2429623fc62c80c7a9239b2bbbfaab227fdc9294a9"} Jul 17 17:57:07.402161 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.402090 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" event={"ID":"a78443b7-6add-4ea0-9b24-949b0b9d117a","Type":"ContainerStarted","Data":"ae07305d9701e0d7bca11bf26c9b91e75d7bc42bfde4a7ad89f87ec4161940ed"} Jul 17 17:57:07.403262 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.403243 2572 generic.go:356] "Generic (PLEG): container finished" podID="19dad044-c54d-4bad-adf5-dcf705460cac" containerID="ac29d80cb1bc2e46b5addf80d656a1346c2829413c7062108bd4c7a13b46ffec" exitCode=0 Jul 17 17:57:07.403346 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.403305 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sz4tq" event={"ID":"19dad044-c54d-4bad-adf5-dcf705460cac","Type":"ContainerDied","Data":"ac29d80cb1bc2e46b5addf80d656a1346c2829413c7062108bd4c7a13b46ffec"} Jul 17 17:57:07.404624 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.404606 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xbw8z" event={"ID":"48aa1cde-e776-496a-9816-9a67614f0210","Type":"ContainerStarted","Data":"23c714d9013725c6830e0a955393ff19b07b3cebf9a891c17f710dcd875b22be"} Jul 17 17:57:07.405844 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.405817 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nwszv" event={"ID":"2e007a42-9178-41c7-bd04-0544e3996bce","Type":"ContainerStarted","Data":"ebcced1365ad3440b146b3de631e262de5e8387ec7b4c02b2b68ba9023335778"} Jul 17 17:57:07.408434 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.408400 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-62.ec2.internal" podStartSLOduration=21.408388503 podStartE2EDuration="21.408388503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:56:50.375962242 +0000 UTC m=+5.785389871" watchObservedRunningTime="2026-07-17 17:57:07.408388503 +0000 UTC m=+22.817816109" Jul 17 17:57:07.408693 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.408672 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hfrgj" podStartSLOduration=3.783035506 podStartE2EDuration="22.408665582s" totalImagesPullingTime="18.625630076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:07.408322806 +0000 UTC m=+22.817750411" watchObservedRunningTime="2026-07-17 17:57:07.408665582 +0000 UTC m=+22.818093187" Jul 17 17:57:07.422168 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.422135 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gpnwp" podStartSLOduration=3.799388343 podStartE2EDuration="22.422122573s" totalImagesPullingTime="18.62273423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:07.421940074 +0000 UTC m=+22.831367681" watchObservedRunningTime="2026-07-17 17:57:07.422122573 +0000 UTC m=+22.831550178" Jul 17 17:57:07.458662 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.458626 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-m8chw" podStartSLOduration=2.767217294 podStartE2EDuration="21.458612316s" totalImagesPullingTime="18.691395022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:07.458452106 +0000 UTC m=+22.867879712" watchObservedRunningTime="2026-07-17 17:57:07.458612316 +0000 UTC m=+22.868039922" Jul 17 17:57:07.472789 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:07.472754 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xbw8z" podStartSLOduration=8.16893451 podStartE2EDuration="22.472741509s" totalImagesPullingTime="14.303806999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:07.472566337 +0000 UTC m=+22.881993943" watchObservedRunningTime="2026-07-17 17:57:07.472741509 +0000 UTC m=+22.882169114" Jul 17 17:57:08.220106 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:08.219995 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:08.220658 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:08.220151 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:57:08.411423 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:08.411383 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" event={"ID":"a78443b7-6add-4ea0-9b24-949b0b9d117a","Type":"ContainerStarted","Data":"b3a2d4ec4ceb7ec6646a992d04d0c91b7681c2f9b7d235af568956469353a825"} Jul 17 17:57:08.411608 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:08.411481 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" event={"ID":"a78443b7-6add-4ea0-9b24-949b0b9d117a","Type":"ContainerStarted","Data":"4cb6a5ec2305f4976f1fbca5af041ba88168e45750d006568cd762d7198f6879"} Jul 17 17:57:08.413467 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:08.413029 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gqftj" event={"ID":"fefc9465-ec24-48f7-9a99-d6fde7791695","Type":"ContainerStarted","Data":"ecf1867a382d32c508afe13d0627a99c19f2bc1a3856377836f845582c03cffb"} Jul 17 17:57:08.430602 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:08.430550 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gqftj" podStartSLOduration=3.799512426 podStartE2EDuration="22.430530737s" totalImagesPullingTime="18.631018311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:08.430431981 +0000 UTC m=+23.839859589" watchObservedRunningTime="2026-07-17 17:57:08.430530737 +0000 UTC m=+23.839958344" Jul 17 17:57:08.431215 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:08.431177 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nwszv" podStartSLOduration=4.806793121 podStartE2EDuration="23.431162164s" totalImagesPullingTime="18.624369043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:07.489908401 +0000 UTC m=+22.899336005" watchObservedRunningTime="2026-07-17 17:57:08.431162164 +0000 UTC m=+23.840589773" Jul 17 17:57:08.670417 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:08.670389 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xbw8z" Jul 17 17:57:08.670941 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:08.670923 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xbw8z" Jul 17 17:57:08.800067 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:08.800036 2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 17 17:57:09.163243 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:09.163147 2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T17:57:08.800061537Z","UUID":"9c1cd617-3a9d-4f23-93c1-1d1bb51d4e3e","Handler":null,"Name":"","Endpoint":""} Jul 17 17:57:09.165311 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:09.165284 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 17 17:57:09.165505 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:09.165331 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 17 17:57:09.219572 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:09.219538 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:09.219738 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:09.219538 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:09.219738 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:09.219654 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:57:09.219858 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:09.219761 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:57:09.417546 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:09.417459 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" event={"ID":"0bc2b05e-ac19-449a-b861-d729822c0afa","Type":"ContainerStarted","Data":"867069daddf3915a324518121a116c0f54f530330307c2f6d44f1edfa36d3f06"} Jul 17 17:57:10.219073 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:10.219027 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:10.219246 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:10.219161 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:57:10.419390 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:10.419362 2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 17 17:57:11.220153 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:11.220118 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:11.220313 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:11.220119 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:11.220313 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:11.220251 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:57:11.220418 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:11.220342 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:57:12.219976 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:12.219825 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:12.220445 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:12.220077 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:57:12.425316 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:12.425281 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" event={"ID":"0bc2b05e-ac19-449a-b861-d729822c0afa","Type":"ContainerStarted","Data":"7817fcadec947cef391c1306a570369db73dd98a818608346e434aa08655afda"} Jul 17 17:57:12.428351 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:12.428300 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" event={"ID":"a78443b7-6add-4ea0-9b24-949b0b9d117a","Type":"ContainerStarted","Data":"d12d5269159c6e7e12a5e0db437a13a7c11a065c4ea8e235d0ba35347aaac32b"} Jul 17 17:57:12.430148 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:12.430126 2572 generic.go:356] "Generic (PLEG): container finished" podID="19dad044-c54d-4bad-adf5-dcf705460cac" containerID="e46c78ab1453359a5aeefa550ae6f9c6d26ce52a6b53c041d18342204ad1369a" exitCode=0 Jul 17 17:57:12.430263 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:12.430156 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sz4tq" event={"ID":"19dad044-c54d-4bad-adf5-dcf705460cac","Type":"ContainerDied","Data":"e46c78ab1453359a5aeefa550ae6f9c6d26ce52a6b53c041d18342204ad1369a"} Jul 17 17:57:12.441467 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:12.441428 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hh5r2" podStartSLOduration=4.33137414 podStartE2EDuration="27.441405411s" totalImagesPullingTime="23.110031271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:56:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:12.441360702 +0000 UTC m=+27.850788434" watchObservedRunningTime="2026-07-17 17:57:12.441405411 +0000 UTC m=+27.850833014" Jul 17 17:57:13.219179 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:13.219144 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:13.219464 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:13.219144 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:13.219464 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:13.219282 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:57:13.219464 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:13.219313 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:57:14.219882 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:14.219624 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:14.220417 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:14.219891 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:57:14.437928 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:14.437884 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" event={"ID":"a78443b7-6add-4ea0-9b24-949b0b9d117a","Type":"ContainerStarted","Data":"9f48495a268ae21754c706b1e1b73e3c76e2a2204694318762660c2fe7b0ca53"} Jul 17 17:57:14.438325 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:14.438239 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:57:14.440003 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:14.439970 2572 generic.go:356] "Generic (PLEG): container finished" podID="19dad044-c54d-4bad-adf5-dcf705460cac" containerID="572f53b5e555022883be692097086e54147349cf47a4d0125fe581a0869d1c5d" exitCode=0 Jul 17 17:57:14.440128 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:14.440028 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sz4tq" event={"ID":"19dad044-c54d-4bad-adf5-dcf705460cac","Type":"ContainerDied","Data":"572f53b5e555022883be692097086e54147349cf47a4d0125fe581a0869d1c5d"} Jul 17 17:57:14.454611 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:14.454592 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:57:14.464535 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:14.464497 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" podStartSLOduration=10.74544805 podStartE2EDuration="29.464484425s" totalImagesPullingTime="18.719036375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:14.463096696 +0000 UTC m=+29.872524321" watchObservedRunningTime="2026-07-17 17:57:14.464484425 +0000 UTC m=+29.873912033" Jul 17 17:57:15.070810 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:15.070784 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xbw8z" Jul 17 17:57:15.070971 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:15.070884 2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 17 17:57:15.071274 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:15.071259 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xbw8z" Jul 17 17:57:15.219790 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:15.219757 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:15.219945 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:15.219903 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:57:15.219945 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:15.219943 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:15.220302 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:15.220043 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:57:15.442427 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:15.442352 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:57:15.442427 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:15.442429 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:57:15.456924 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:15.456894 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:57:16.219779 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:16.219750 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:16.219924 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:16.219856 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:57:16.446779 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:16.446745 2572 generic.go:356] "Generic (PLEG): container finished" podID="19dad044-c54d-4bad-adf5-dcf705460cac" containerID="7da843092aab5a2052b352ba07bb92816c8d852adf10f9fc0b5cada610191ca5" exitCode=0 Jul 17 17:57:16.447205 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:16.446843 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sz4tq" event={"ID":"19dad044-c54d-4bad-adf5-dcf705460cac","Type":"ContainerDied","Data":"7da843092aab5a2052b352ba07bb92816c8d852adf10f9fc0b5cada610191ca5"} Jul 17 17:57:17.219784 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:17.219741 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:17.219784 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:17.219770 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:17.220000 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:17.219947 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:57:17.220000 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:17.219931 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:57:18.219105 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:18.219066 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:18.219571 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:18.219187 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:57:18.966056 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:18.965997 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltg2b\" (UniqueName: \"kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b\") pod \"network-check-target-wcf2q\" (UID: \"27188dfc-708b-462f-9a9a-24fda6997009\") " pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:18.966247 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:18.966066 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret\") pod \"global-pull-secret-syncer-ts454\" (UID: \"118b7937-fc4d-48fa-927e-2368a2317850\") " pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:18.966247 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:18.966111 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs\") pod \"network-metrics-daemon-5ccsk\" (UID: \"e05c7832-1be4-46bd-91e4-2dd5c8abeb02\") " pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:18.966247 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:18.966178 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:57:18.966247 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:18.966204 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:57:18.966247 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:18.966211 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:57:18.966247 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:18.966218 2572 projected.go:196] Error preparing data for projected volume kube-api-access-ltg2b for pod openshift-network-diagnostics/network-check-target-wcf2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:57:18.966505 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:18.966267 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs podName:e05c7832-1be4-46bd-91e4-2dd5c8abeb02 nodeName:}" failed. No retries permitted until 2026-07-17 17:57:50.96624956 +0000 UTC m=+66.375677144 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs") pod "network-metrics-daemon-5ccsk" (UID: "e05c7832-1be4-46bd-91e4-2dd5c8abeb02") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:57:18.966505 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:18.966281 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b podName:27188dfc-708b-462f-9a9a-24fda6997009 nodeName:}" failed. No retries permitted until 2026-07-17 17:57:50.966274492 +0000 UTC m=+66.375702076 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ltg2b" (UniqueName: "kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b") pod "network-check-target-wcf2q" (UID: "27188dfc-708b-462f-9a9a-24fda6997009") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:57:18.966505 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:18.966312 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:57:18.966505 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:18.966372 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret podName:118b7937-fc4d-48fa-927e-2368a2317850 nodeName:}" failed. No retries permitted until 2026-07-17 17:57:50.966355047 +0000 UTC m=+66.375782632 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret") pod "global-pull-secret-syncer-ts454" (UID: "118b7937-fc4d-48fa-927e-2368a2317850") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:57:19.220197 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:19.220120 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:19.220608 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:19.220120 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:19.220608 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:19.220258 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:57:19.220608 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:19.220341 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:57:20.219019 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:20.218975 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:20.219174 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:20.219122 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:57:21.219489 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:21.219455 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:21.219867 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:21.219572 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:57:21.219867 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:21.219624 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:21.219867 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:21.219706 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:57:21.464836 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:21.462471 2572 generic.go:356] "Generic (PLEG): container finished" podID="19dad044-c54d-4bad-adf5-dcf705460cac" containerID="bfee34a01a57365c613d6368446d6d48df3825dae6403d0037b2afcbfc62df65" exitCode=0 Jul 17 17:57:21.464836 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:21.462526 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sz4tq" event={"ID":"19dad044-c54d-4bad-adf5-dcf705460cac","Type":"ContainerDied","Data":"bfee34a01a57365c613d6368446d6d48df3825dae6403d0037b2afcbfc62df65"} Jul 17 17:57:22.219994 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:22.219961 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:22.220367 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:22.220073 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:57:22.467739 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:22.467706 2572 generic.go:356] "Generic (PLEG): container finished" podID="19dad044-c54d-4bad-adf5-dcf705460cac" containerID="d67778689fbdc729c87dd593478a054b78188a58e0b7c93c088d7a344f5abd91" exitCode=0 Jul 17 17:57:22.467899 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:22.467755 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sz4tq" event={"ID":"19dad044-c54d-4bad-adf5-dcf705460cac","Type":"ContainerDied","Data":"d67778689fbdc729c87dd593478a054b78188a58e0b7c93c088d7a344f5abd91"} Jul 17 17:57:23.219688 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:23.219500 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:23.219862 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:23.219500 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:23.219862 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:23.219794 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:57:23.219862 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:23.219828 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:57:23.473148 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:23.473045 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sz4tq" event={"ID":"19dad044-c54d-4bad-adf5-dcf705460cac","Type":"ContainerStarted","Data":"36e7e8c920cd922107dfc0c060116001d1dcfdb7e8f80332ce5517ed966d2192"} Jul 17 17:57:23.496195 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:23.496147 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sz4tq" podStartSLOduration=7.953922722 podStartE2EDuration="37.496132619s" totalImagesPullingTime="29.542209897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:56:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:23.495086763 +0000 UTC m=+38.904514368" watchObservedRunningTime="2026-07-17 17:57:23.496132619 +0000 UTC m=+38.905560203" Jul 17 17:57:24.219548 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:24.219519 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:24.219744 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:24.219611 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:57:25.220372 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:25.220339 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:25.220772 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:25.220417 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:57:25.220772 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:25.220507 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:25.220772 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:25.220594 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:57:26.219088 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:26.219060 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:26.219259 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:26.219158 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:57:27.219401 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:27.219364 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:27.219854 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:27.219364 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:27.219854 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:27.219502 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:57:27.219854 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:27.219566 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:57:28.219546 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:28.219208 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:28.219546 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:28.219337 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:57:28.889310 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:28.889276 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ts454"] Jul 17 17:57:28.889456 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:28.889392 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:28.889542 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:28.889518 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:57:28.895205 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:28.895173 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wcf2q"] Jul 17 17:57:28.895357 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:28.895289 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:28.895430 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:28.895401 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:57:28.895881 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:28.895860 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5ccsk"] Jul 17 17:57:28.895996 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:28.895957 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:28.896095 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:28.896071 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:57:30.219751 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:30.219717 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:30.220296 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:30.219761 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:30.220296 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:30.219817 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:30.220296 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:30.219836 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:57:30.220296 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:30.219888 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:57:30.220296 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:30.219947 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:57:32.220017 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.219988 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:32.220390 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.220123 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:32.220390 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.220135 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:32.220390 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:32.220139 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ts454" podUID="118b7937-fc4d-48fa-927e-2368a2317850" Jul 17 17:57:32.220390 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:32.220194 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcf2q" podUID="27188dfc-708b-462f-9a9a-24fda6997009" Jul 17 17:57:32.220390 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:32.220249 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5ccsk" podUID="e05c7832-1be4-46bd-91e4-2dd5c8abeb02" Jul 17 17:57:32.539968 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.539948 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-62.ec2.internal" event="NodeReady" Jul 17 17:57:32.540132 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.540086 2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 17 17:57:32.572786 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.572757 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b558bd496-vjjrw"] Jul 17 17:57:32.587024 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.586970 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.588278 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.588251 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nxcs2"] Jul 17 17:57:32.589634 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.589612 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 17 17:57:32.589761 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.589615 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 17 17:57:32.589761 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.589690 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 17 17:57:32.589912 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.589898 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c22cb\"" Jul 17 17:57:32.596543 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.596524 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 17 17:57:32.602547 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.602529 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b558bd496-vjjrw"] Jul 17 17:57:32.602547 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.602557 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nk6r6"] Jul 17 17:57:32.602680 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.602661 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nxcs2" Jul 17 17:57:32.605322 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.605216 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 17:57:32.605322 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.605298 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s6gdc\"" Jul 17 17:57:32.605496 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.605356 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 17:57:32.618700 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.618683 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nxcs2"] Jul 17 17:57:32.618783 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.618732 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nk6r6"] Jul 17 17:57:32.618817 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.618786 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nk6r6" Jul 17 17:57:32.620963 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.620946 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 17:57:32.621113 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.621091 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 17:57:32.621158 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.621119 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hnrcx\"" Jul 17 17:57:32.621199 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.621186 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 17:57:32.662385 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.662363 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88632bb0-62df-4983-970b-ac75701af480-installation-pull-secrets\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.662472 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.662388 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbkg2\" (UniqueName: \"kubernetes.io/projected/88632bb0-62df-4983-970b-ac75701af480-kube-api-access-vbkg2\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.662472 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.662407 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88632bb0-62df-4983-970b-ac75701af480-trusted-ca\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.662472 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.662422 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ead021cb-e13c-433e-b94f-de93c3165044-cert\") pod \"ingress-canary-nk6r6\" (UID: \"ead021cb-e13c-433e-b94f-de93c3165044\") " pod="openshift-ingress-canary/ingress-canary-nk6r6" Jul 17 17:57:32.662472 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.662437 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2z4d\" (UniqueName: \"kubernetes.io/projected/ead021cb-e13c-433e-b94f-de93c3165044-kube-api-access-x2z4d\") pod \"ingress-canary-nk6r6\" (UID: \"ead021cb-e13c-433e-b94f-de93c3165044\") " pod="openshift-ingress-canary/ingress-canary-nk6r6" Jul 17 17:57:32.662597 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.662472 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eb2fc23-38d4-4bac-874b-706356e5f1f7-metrics-tls\") pod \"dns-default-nxcs2\" (UID: \"9eb2fc23-38d4-4bac-874b-706356e5f1f7\") " pod="openshift-dns/dns-default-nxcs2" Jul 17 17:57:32.662597 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.662495 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb2fc23-38d4-4bac-874b-706356e5f1f7-tmp-dir\") pod \"dns-default-nxcs2\" (UID: \"9eb2fc23-38d4-4bac-874b-706356e5f1f7\") " pod="openshift-dns/dns-default-nxcs2" Jul 17 17:57:32.662597 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.662517 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88632bb0-62df-4983-970b-ac75701af480-bound-sa-token\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.662682 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.662593 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t8cs\" (UniqueName: \"kubernetes.io/projected/9eb2fc23-38d4-4bac-874b-706356e5f1f7-kube-api-access-5t8cs\") pod \"dns-default-nxcs2\" (UID: \"9eb2fc23-38d4-4bac-874b-706356e5f1f7\") " pod="openshift-dns/dns-default-nxcs2" Jul 17 17:57:32.662682 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.662630 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb2fc23-38d4-4bac-874b-706356e5f1f7-config-volume\") pod \"dns-default-nxcs2\" (UID: \"9eb2fc23-38d4-4bac-874b-706356e5f1f7\") " pod="openshift-dns/dns-default-nxcs2" Jul 17 17:57:32.662682 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.662654 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88632bb0-62df-4983-970b-ac75701af480-image-registry-private-configuration\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.662682 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.662675 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88632bb0-62df-4983-970b-ac75701af480-registry-tls\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.662797 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.662699 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88632bb0-62df-4983-970b-ac75701af480-ca-trust-extracted\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.662797 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.662719 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88632bb0-62df-4983-970b-ac75701af480-registry-certificates\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.692255 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.692230 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-g6q7x"] Jul 17 17:57:32.698637 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.698624 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g6q7x" Jul 17 17:57:32.701250 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.701234 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 17:57:32.701453 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.701432 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 17:57:32.701554 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.701455 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 17:57:32.701554 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.701455 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 17:57:32.701554 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.701463 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ns7ff\"" Jul 17 17:57:32.703885 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.703866 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g6q7x"] Jul 17 17:57:32.763590 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.763557 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88632bb0-62df-4983-970b-ac75701af480-image-registry-private-configuration\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.763590 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.763591 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88632bb0-62df-4983-970b-ac75701af480-registry-tls\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.763768 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.763612 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88632bb0-62df-4983-970b-ac75701af480-ca-trust-extracted\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.763768 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.763637 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88632bb0-62df-4983-970b-ac75701af480-registry-certificates\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.763768 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.763679 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88632bb0-62df-4983-970b-ac75701af480-installation-pull-secrets\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.763768 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.763701 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbkg2\" (UniqueName: \"kubernetes.io/projected/88632bb0-62df-4983-970b-ac75701af480-kube-api-access-vbkg2\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.763768 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.763728 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76665724-fb0c-43fe-b09d-e3d43ede2fc0-crio-socket\") pod \"insights-runtime-extractor-g6q7x\" (UID: \"76665724-fb0c-43fe-b09d-e3d43ede2fc0\") " pod="openshift-insights/insights-runtime-extractor-g6q7x" Jul 17 17:57:32.763768 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.763768 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88632bb0-62df-4983-970b-ac75701af480-trusted-ca\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.764045 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.763917 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ead021cb-e13c-433e-b94f-de93c3165044-cert\") pod \"ingress-canary-nk6r6\" (UID: \"ead021cb-e13c-433e-b94f-de93c3165044\") " pod="openshift-ingress-canary/ingress-canary-nk6r6" Jul 17 17:57:32.764287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.764043 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/88632bb0-62df-4983-970b-ac75701af480-ca-trust-extracted\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.764287 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.764059 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2z4d\" (UniqueName: \"kubernetes.io/projected/ead021cb-e13c-433e-b94f-de93c3165044-kube-api-access-x2z4d\") pod \"ingress-canary-nk6r6\" (UID: \"ead021cb-e13c-433e-b94f-de93c3165044\") " pod="openshift-ingress-canary/ingress-canary-nk6r6" Jul 17 17:57:32.764771 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.764317 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eb2fc23-38d4-4bac-874b-706356e5f1f7-metrics-tls\") pod \"dns-default-nxcs2\" (UID: \"9eb2fc23-38d4-4bac-874b-706356e5f1f7\") " pod="openshift-dns/dns-default-nxcs2" Jul 17 17:57:32.764771 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.764345 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb2fc23-38d4-4bac-874b-706356e5f1f7-tmp-dir\") pod \"dns-default-nxcs2\" (UID: \"9eb2fc23-38d4-4bac-874b-706356e5f1f7\") " pod="openshift-dns/dns-default-nxcs2" Jul 17 17:57:32.764771 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.764372 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88632bb0-62df-4983-970b-ac75701af480-bound-sa-token\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.764771 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.764445 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8cs\" (UniqueName: \"kubernetes.io/projected/9eb2fc23-38d4-4bac-874b-706356e5f1f7-kube-api-access-5t8cs\") pod \"dns-default-nxcs2\" (UID: \"9eb2fc23-38d4-4bac-874b-706356e5f1f7\") " pod="openshift-dns/dns-default-nxcs2" Jul 17 17:57:32.764771 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.764495 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76665724-fb0c-43fe-b09d-e3d43ede2fc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g6q7x\" (UID: \"76665724-fb0c-43fe-b09d-e3d43ede2fc0\") " pod="openshift-insights/insights-runtime-extractor-g6q7x" Jul 17 17:57:32.764771 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.764526 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76665724-fb0c-43fe-b09d-e3d43ede2fc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g6q7x\" (UID: \"76665724-fb0c-43fe-b09d-e3d43ede2fc0\") " pod="openshift-insights/insights-runtime-extractor-g6q7x" Jul 17 17:57:32.764771 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.764559 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb2fc23-38d4-4bac-874b-706356e5f1f7-config-volume\") pod \"dns-default-nxcs2\" (UID: \"9eb2fc23-38d4-4bac-874b-706356e5f1f7\") " pod="openshift-dns/dns-default-nxcs2" Jul 17 17:57:32.764771 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.764582 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76665724-fb0c-43fe-b09d-e3d43ede2fc0-data-volume\") pod \"insights-runtime-extractor-g6q7x\" (UID: \"76665724-fb0c-43fe-b09d-e3d43ede2fc0\") " pod="openshift-insights/insights-runtime-extractor-g6q7x" Jul 17 17:57:32.764771 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.764608 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcvxj\" (UniqueName: \"kubernetes.io/projected/76665724-fb0c-43fe-b09d-e3d43ede2fc0-kube-api-access-zcvxj\") pod \"insights-runtime-extractor-g6q7x\" (UID: \"76665724-fb0c-43fe-b09d-e3d43ede2fc0\") " pod="openshift-insights/insights-runtime-extractor-g6q7x" Jul 17 17:57:32.764771 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.764669 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb2fc23-38d4-4bac-874b-706356e5f1f7-tmp-dir\") pod \"dns-default-nxcs2\" (UID: \"9eb2fc23-38d4-4bac-874b-706356e5f1f7\") " pod="openshift-dns/dns-default-nxcs2" Jul 17 17:57:32.764771 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.764737 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/88632bb0-62df-4983-970b-ac75701af480-registry-certificates\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.765268 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.764893 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/88632bb0-62df-4983-970b-ac75701af480-trusted-ca\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.765268 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.765223 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9eb2fc23-38d4-4bac-874b-706356e5f1f7-config-volume\") pod \"dns-default-nxcs2\" (UID: \"9eb2fc23-38d4-4bac-874b-706356e5f1f7\") " pod="openshift-dns/dns-default-nxcs2" Jul 17 17:57:32.768183 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.768152 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/88632bb0-62df-4983-970b-ac75701af480-installation-pull-secrets\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.768183 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.768168 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ead021cb-e13c-433e-b94f-de93c3165044-cert\") pod \"ingress-canary-nk6r6\" (UID: \"ead021cb-e13c-433e-b94f-de93c3165044\") " pod="openshift-ingress-canary/ingress-canary-nk6r6" Jul 17 17:57:32.768310 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.768180 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/88632bb0-62df-4983-970b-ac75701af480-registry-tls\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.768310 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.768171 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/88632bb0-62df-4983-970b-ac75701af480-image-registry-private-configuration\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.768310 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.768212 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9eb2fc23-38d4-4bac-874b-706356e5f1f7-metrics-tls\") pod \"dns-default-nxcs2\" (UID: \"9eb2fc23-38d4-4bac-874b-706356e5f1f7\") " pod="openshift-dns/dns-default-nxcs2" Jul 17 17:57:32.771613 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.771560 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88632bb0-62df-4983-970b-ac75701af480-bound-sa-token\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.771896 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.771876 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2z4d\" (UniqueName: \"kubernetes.io/projected/ead021cb-e13c-433e-b94f-de93c3165044-kube-api-access-x2z4d\") pod \"ingress-canary-nk6r6\" (UID: \"ead021cb-e13c-433e-b94f-de93c3165044\") " pod="openshift-ingress-canary/ingress-canary-nk6r6" Jul 17 17:57:32.771962 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.771944 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8cs\" (UniqueName: \"kubernetes.io/projected/9eb2fc23-38d4-4bac-874b-706356e5f1f7-kube-api-access-5t8cs\") pod \"dns-default-nxcs2\" (UID: \"9eb2fc23-38d4-4bac-874b-706356e5f1f7\") " pod="openshift-dns/dns-default-nxcs2" Jul 17 17:57:32.772334 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.772315 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbkg2\" (UniqueName: \"kubernetes.io/projected/88632bb0-62df-4983-970b-ac75701af480-kube-api-access-vbkg2\") pod \"image-registry-7b558bd496-vjjrw\" (UID: \"88632bb0-62df-4983-970b-ac75701af480\") " pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.865641 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.865520 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76665724-fb0c-43fe-b09d-e3d43ede2fc0-data-volume\") pod \"insights-runtime-extractor-g6q7x\" (UID: \"76665724-fb0c-43fe-b09d-e3d43ede2fc0\") " pod="openshift-insights/insights-runtime-extractor-g6q7x" Jul 17 17:57:32.865641 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.865564 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcvxj\" (UniqueName: \"kubernetes.io/projected/76665724-fb0c-43fe-b09d-e3d43ede2fc0-kube-api-access-zcvxj\") pod \"insights-runtime-extractor-g6q7x\" (UID: \"76665724-fb0c-43fe-b09d-e3d43ede2fc0\") " pod="openshift-insights/insights-runtime-extractor-g6q7x" Jul 17 17:57:32.865641 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.865631 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76665724-fb0c-43fe-b09d-e3d43ede2fc0-crio-socket\") pod \"insights-runtime-extractor-g6q7x\" (UID: \"76665724-fb0c-43fe-b09d-e3d43ede2fc0\") " pod="openshift-insights/insights-runtime-extractor-g6q7x" Jul 17 17:57:32.865904 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.865711 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76665724-fb0c-43fe-b09d-e3d43ede2fc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g6q7x\" (UID: \"76665724-fb0c-43fe-b09d-e3d43ede2fc0\") " pod="openshift-insights/insights-runtime-extractor-g6q7x" Jul 17 17:57:32.865904 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.865733 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76665724-fb0c-43fe-b09d-e3d43ede2fc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g6q7x\" (UID: \"76665724-fb0c-43fe-b09d-e3d43ede2fc0\") " pod="openshift-insights/insights-runtime-extractor-g6q7x" Jul 17 17:57:32.866037 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.865921 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/76665724-fb0c-43fe-b09d-e3d43ede2fc0-crio-socket\") pod \"insights-runtime-extractor-g6q7x\" (UID: \"76665724-fb0c-43fe-b09d-e3d43ede2fc0\") " pod="openshift-insights/insights-runtime-extractor-g6q7x" Jul 17 17:57:32.866037 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.865968 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/76665724-fb0c-43fe-b09d-e3d43ede2fc0-data-volume\") pod \"insights-runtime-extractor-g6q7x\" (UID: \"76665724-fb0c-43fe-b09d-e3d43ede2fc0\") " pod="openshift-insights/insights-runtime-extractor-g6q7x" Jul 17 17:57:32.866244 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.866221 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/76665724-fb0c-43fe-b09d-e3d43ede2fc0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g6q7x\" (UID: \"76665724-fb0c-43fe-b09d-e3d43ede2fc0\") " pod="openshift-insights/insights-runtime-extractor-g6q7x" Jul 17 17:57:32.868242 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.868218 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/76665724-fb0c-43fe-b09d-e3d43ede2fc0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g6q7x\" (UID: \"76665724-fb0c-43fe-b09d-e3d43ede2fc0\") " pod="openshift-insights/insights-runtime-extractor-g6q7x" Jul 17 17:57:32.873388 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.873367 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcvxj\" (UniqueName: \"kubernetes.io/projected/76665724-fb0c-43fe-b09d-e3d43ede2fc0-kube-api-access-zcvxj\") pod \"insights-runtime-extractor-g6q7x\" (UID: \"76665724-fb0c-43fe-b09d-e3d43ede2fc0\") " pod="openshift-insights/insights-runtime-extractor-g6q7x" Jul 17 17:57:32.898260 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.898238 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:32.912268 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.912250 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nxcs2" Jul 17 17:57:32.927188 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:32.927167 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nk6r6" Jul 17 17:57:33.007219 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:33.007193 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g6q7x" Jul 17 17:57:33.093386 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:33.093358 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nxcs2"] Jul 17 17:57:33.098263 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:33.098166 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b558bd496-vjjrw"] Jul 17 17:57:33.101218 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:57:33.100997 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88632bb0_62df_4983_970b_ac75701af480.slice/crio-4b9acea62ddb3190ec31ca0044f09e0b1414dce6c7eeabccdd5466f0d0839fe1 WatchSource:0}: Error finding container 4b9acea62ddb3190ec31ca0044f09e0b1414dce6c7eeabccdd5466f0d0839fe1: Status 404 returned error can't find the container with id 4b9acea62ddb3190ec31ca0044f09e0b1414dce6c7eeabccdd5466f0d0839fe1 Jul 17 17:57:33.125783 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:33.125722 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nk6r6"] Jul 17 17:57:33.135371 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:57:33.135341 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead021cb_e13c_433e_b94f_de93c3165044.slice/crio-b37207be6381cbb1136f4e2bf3c75f39cdff655135f3a07956d96e3a432cec44 WatchSource:0}: Error finding container b37207be6381cbb1136f4e2bf3c75f39cdff655135f3a07956d96e3a432cec44: Status 404 returned error can't find the container with id b37207be6381cbb1136f4e2bf3c75f39cdff655135f3a07956d96e3a432cec44 Jul 17 17:57:33.165795 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:33.165769 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g6q7x"] Jul 17 17:57:33.169138 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:57:33.169114 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76665724_fb0c_43fe_b09d_e3d43ede2fc0.slice/crio-2fbdd36e06b2ed62913eca7a4f7fe7b9df963fd0a1373b8380d0cc4aad9258ca WatchSource:0}: Error finding container 2fbdd36e06b2ed62913eca7a4f7fe7b9df963fd0a1373b8380d0cc4aad9258ca: Status 404 returned error can't find the container with id 2fbdd36e06b2ed62913eca7a4f7fe7b9df963fd0a1373b8380d0cc4aad9258ca Jul 17 17:57:33.497089 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:33.496992 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nxcs2" event={"ID":"9eb2fc23-38d4-4bac-874b-706356e5f1f7","Type":"ContainerStarted","Data":"87e38e7fc4802cb5566b38b2504c88c510d2774575d6a2b09643c95e3d7773e9"} Jul 17 17:57:33.498311 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:33.498283 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g6q7x" event={"ID":"76665724-fb0c-43fe-b09d-e3d43ede2fc0","Type":"ContainerStarted","Data":"2fbdd36e06b2ed62913eca7a4f7fe7b9df963fd0a1373b8380d0cc4aad9258ca"} Jul 17 17:57:33.499372 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:33.499353 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nk6r6" event={"ID":"ead021cb-e13c-433e-b94f-de93c3165044","Type":"ContainerStarted","Data":"b37207be6381cbb1136f4e2bf3c75f39cdff655135f3a07956d96e3a432cec44"} Jul 17 17:57:33.500789 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:33.500768 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" event={"ID":"88632bb0-62df-4983-970b-ac75701af480","Type":"ContainerStarted","Data":"c0f01c215b9929921b7ca579b848b8747b2c303a367781bdbd77fc8f1e3cff8b"} Jul 17 17:57:33.500882 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:33.500802 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" event={"ID":"88632bb0-62df-4983-970b-ac75701af480","Type":"ContainerStarted","Data":"4b9acea62ddb3190ec31ca0044f09e0b1414dce6c7eeabccdd5466f0d0839fe1"} Jul 17 17:57:33.500935 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:33.500883 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:57:33.519696 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:33.519564 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" podStartSLOduration=6.519546089 podStartE2EDuration="6.519546089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:33.519274345 +0000 UTC m=+48.928701962" watchObservedRunningTime="2026-07-17 17:57:33.519546089 +0000 UTC m=+48.928973696" Jul 17 17:57:34.219836 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:34.219746 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:34.219836 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:34.219771 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:34.219836 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:34.219771 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:34.222813 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:34.222790 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 17:57:34.225126 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:34.225107 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dn68c\"" Jul 17 17:57:34.225329 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:34.225315 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 17:57:34.225535 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:34.225520 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 17:57:34.225713 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:34.225699 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jb7qg\"" Jul 17 17:57:34.226820 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:34.225899 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 17:57:39.520462 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:39.520228 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nk6r6" event={"ID":"ead021cb-e13c-433e-b94f-de93c3165044","Type":"ContainerStarted","Data":"a2a3aca57878cd830a08840a74c8c76840ed12500272e6dbf93fa7dd27a44222"} Jul 17 17:57:39.523416 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:39.522981 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nxcs2" event={"ID":"9eb2fc23-38d4-4bac-874b-706356e5f1f7","Type":"ContainerStarted","Data":"6946f51d15b58d39e83746c6bc1433c3f79a61c5243ffde517d92d422d7ae9a1"} Jul 17 17:57:40.528788 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:40.528508 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nxcs2" event={"ID":"9eb2fc23-38d4-4bac-874b-706356e5f1f7","Type":"ContainerStarted","Data":"9fbff12155c44d844bf00376f8659f11f2ea742f66128639e26c4356ab548457"} Jul 17 17:57:40.528788 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:40.528605 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nxcs2" Jul 17 17:57:40.530724 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:40.530685 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g6q7x" event={"ID":"76665724-fb0c-43fe-b09d-e3d43ede2fc0","Type":"ContainerStarted","Data":"c255e74625b7b4a6f60e79859008c6afddb0b4d1791d0299c14464195ab70e77"} Jul 17 17:57:40.546930 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:40.545943 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nxcs2" podStartSLOduration=2.337566357 podStartE2EDuration="8.545923988s" totalImagesPullingTime="6.208357631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:40.545871032 +0000 UTC m=+55.955298663" watchObservedRunningTime="2026-07-17 17:57:40.545923988 +0000 UTC m=+55.955351597" Jul 17 17:57:40.546930 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:40.546120 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nk6r6" podStartSLOduration=2.374197237 podStartE2EDuration="8.546106417s" totalImagesPullingTime="6.17190918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:39.542567476 +0000 UTC m=+54.951995086" watchObservedRunningTime="2026-07-17 17:57:40.546106417 +0000 UTC m=+55.955534024" Jul 17 17:57:42.823659 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:42.823618 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rfp68"] Jul 17 17:57:42.856289 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:42.856261 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:42.859108 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:42.859080 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 17:57:42.859257 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:42.859235 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 17:57:42.859354 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:42.859248 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 17:57:42.860069 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:42.860049 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 17:57:42.860182 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:42.860078 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4h8ql\"" Jul 17 17:57:42.860182 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:42.860104 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 17:57:42.860182 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:42.860130 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 17:57:42.946851 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:42.946809 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f293570-245b-432e-b029-31aa06cd7156-metrics-client-ca\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:42.947048 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:42.946896 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1f293570-245b-432e-b029-31aa06cd7156-node-exporter-textfile\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:42.947048 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:42.946933 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1f293570-245b-432e-b029-31aa06cd7156-node-exporter-wtmp\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:42.947048 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:42.946956 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f293570-245b-432e-b029-31aa06cd7156-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:42.947048 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:42.947034 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1f293570-245b-432e-b029-31aa06cd7156-node-exporter-accelerators-collector-config\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:42.947262 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:42.947088 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2xdn\" (UniqueName: \"kubernetes.io/projected/1f293570-245b-432e-b029-31aa06cd7156-kube-api-access-r2xdn\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:42.947262 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:42.947119 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1f293570-245b-432e-b029-31aa06cd7156-node-exporter-tls\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:42.947262 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:42.947145 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1f293570-245b-432e-b029-31aa06cd7156-root\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:42.947262 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:42.947192 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f293570-245b-432e-b029-31aa06cd7156-sys\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.048616 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.048581 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f293570-245b-432e-b029-31aa06cd7156-sys\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.048797 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.048643 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f293570-245b-432e-b029-31aa06cd7156-metrics-client-ca\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.048797 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.048680 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1f293570-245b-432e-b029-31aa06cd7156-node-exporter-textfile\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.048797 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.048707 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1f293570-245b-432e-b029-31aa06cd7156-node-exporter-wtmp\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.048797 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.048735 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f293570-245b-432e-b029-31aa06cd7156-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.048797 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.048790 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1f293570-245b-432e-b029-31aa06cd7156-node-exporter-accelerators-collector-config\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.049115 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.048837 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2xdn\" (UniqueName: \"kubernetes.io/projected/1f293570-245b-432e-b029-31aa06cd7156-kube-api-access-r2xdn\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.049115 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.048857 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1f293570-245b-432e-b029-31aa06cd7156-node-exporter-tls\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.049115 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.048881 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1f293570-245b-432e-b029-31aa06cd7156-root\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.049115 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.048969 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1f293570-245b-432e-b029-31aa06cd7156-root\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.049115 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.049098 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1f293570-245b-432e-b029-31aa06cd7156-node-exporter-wtmp\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.049115 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.049104 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1f293570-245b-432e-b029-31aa06cd7156-node-exporter-textfile\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.049413 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.048705 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f293570-245b-432e-b029-31aa06cd7156-sys\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.049413 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.049338 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f293570-245b-432e-b029-31aa06cd7156-metrics-client-ca\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.049413 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:43.049369 2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 17 17:57:43.049567 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:43.049463 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f293570-245b-432e-b029-31aa06cd7156-node-exporter-tls podName:1f293570-245b-432e-b029-31aa06cd7156 nodeName:}" failed. No retries permitted until 2026-07-17 17:57:43.549433077 +0000 UTC m=+58.958860678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1f293570-245b-432e-b029-31aa06cd7156-node-exporter-tls") pod "node-exporter-rfp68" (UID: "1f293570-245b-432e-b029-31aa06cd7156") : secret "node-exporter-tls" not found Jul 17 17:57:43.049770 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.049750 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1f293570-245b-432e-b029-31aa06cd7156-node-exporter-accelerators-collector-config\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.053865 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.053837 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f293570-245b-432e-b029-31aa06cd7156-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.057504 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.057479 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2xdn\" (UniqueName: \"kubernetes.io/projected/1f293570-245b-432e-b029-31aa06cd7156-kube-api-access-r2xdn\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.541633 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.541608 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g6q7x" event={"ID":"76665724-fb0c-43fe-b09d-e3d43ede2fc0","Type":"ContainerStarted","Data":"f9aa3e390b6bc72cb567d557f3a79f177abf17814a1038577e9ec22353f2c8f8"} Jul 17 17:57:43.552172 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.552156 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1f293570-245b-432e-b029-31aa06cd7156-node-exporter-tls\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.554264 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.554245 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1f293570-245b-432e-b029-31aa06cd7156-node-exporter-tls\") pod \"node-exporter-rfp68\" (UID: \"1f293570-245b-432e-b029-31aa06cd7156\") " pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.766776 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.766701 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rfp68" Jul 17 17:57:43.775703 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:57:43.775679 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f293570_245b_432e_b029_31aa06cd7156.slice/crio-ea5e2cdcd774b6ea4d737e34a010a0117b0f3a30c1c6617f6e88da7a259c16e5 WatchSource:0}: Error finding container ea5e2cdcd774b6ea4d737e34a010a0117b0f3a30c1c6617f6e88da7a259c16e5: Status 404 returned error can't find the container with id ea5e2cdcd774b6ea4d737e34a010a0117b0f3a30c1c6617f6e88da7a259c16e5 Jul 17 17:57:43.871214 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.871188 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:57:43.873840 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.873824 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:43.876368 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.876343 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 17 17:57:43.876476 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.876343 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 17 17:57:43.876476 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.876464 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 17 17:57:43.876588 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.876476 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 17 17:57:43.876638 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.876591 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 17 17:57:43.876783 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.876732 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 17 17:57:43.877143 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.877110 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 17 17:57:43.877284 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.877157 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 17 17:57:43.877284 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.877172 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 17 17:57:43.877284 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.877195 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gc28g\"" Jul 17 17:57:43.893142 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.893117 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:57:43.955359 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.955325 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:43.955548 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.955376 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:43.955548 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.955415 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c7e5de6-b30e-4800-b952-332c75398e11-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:43.955548 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.955486 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-web-config\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:43.955715 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.955548 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md5jt\" (UniqueName: \"kubernetes.io/projected/9c7e5de6-b30e-4800-b952-332c75398e11-kube-api-access-md5jt\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:43.955715 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.955596 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c7e5de6-b30e-4800-b952-332c75398e11-config-out\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:43.955792 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.955703 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:43.955792 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.955756 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c7e5de6-b30e-4800-b952-332c75398e11-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:43.955869 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.955816 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:43.955869 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.955844 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c7e5de6-b30e-4800-b952-332c75398e11-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:43.955954 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.955880 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:43.955954 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.955913 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-config-volume\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:43.955954 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:43.955937 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c7e5de6-b30e-4800-b952-332c75398e11-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.056739 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.056704 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.056924 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.056764 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c7e5de6-b30e-4800-b952-332c75398e11-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.056924 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.056800 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.056924 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.056825 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c7e5de6-b30e-4800-b952-332c75398e11-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.056924 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.056850 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.056924 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.056881 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-config-volume\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.056924 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.056902 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c7e5de6-b30e-4800-b952-332c75398e11-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.057183 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.056936 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.057183 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.056964 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.057183 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:44.056991 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c7e5de6-b30e-4800-b952-332c75398e11-alertmanager-trusted-ca-bundle podName:9c7e5de6-b30e-4800-b952-332c75398e11 nodeName:}" failed. No retries permitted until 2026-07-17 17:57:44.556968008 +0000 UTC m=+59.966395595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/9c7e5de6-b30e-4800-b952-332c75398e11-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "9c7e5de6-b30e-4800-b952-332c75398e11") : configmap references non-existent config key: ca-bundle.crt Jul 17 17:57:44.057183 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.057063 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c7e5de6-b30e-4800-b952-332c75398e11-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.057183 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.057096 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-web-config\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.057183 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.057124 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md5jt\" (UniqueName: \"kubernetes.io/projected/9c7e5de6-b30e-4800-b952-332c75398e11-kube-api-access-md5jt\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.057183 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.057167 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c7e5de6-b30e-4800-b952-332c75398e11-config-out\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.057959 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.057931 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c7e5de6-b30e-4800-b952-332c75398e11-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.059064 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.058453 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c7e5de6-b30e-4800-b952-332c75398e11-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.059562 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:44.059541 2572 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 17 17:57:44.059653 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:57:44.059607 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-main-tls podName:9c7e5de6-b30e-4800-b952-332c75398e11 nodeName:}" failed. No retries permitted until 2026-07-17 17:57:44.559588985 +0000 UTC m=+59.969016575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "9c7e5de6-b30e-4800-b952-332c75398e11") : secret "alertmanager-main-tls" not found Jul 17 17:57:44.061052 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.060903 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-config-volume\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.061372 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.061345 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c7e5de6-b30e-4800-b952-332c75398e11-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.061725 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.061597 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-web-config\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.061966 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.061943 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.062472 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.062451 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.062472 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.062464 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.062811 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.062779 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.063226 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.063207 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c7e5de6-b30e-4800-b952-332c75398e11-config-out\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.068086 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.068066 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md5jt\" (UniqueName: \"kubernetes.io/projected/9c7e5de6-b30e-4800-b952-332c75398e11-kube-api-access-md5jt\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.392721 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.392631 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-5v6b2"] Jul 17 17:57:44.394659 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.394635 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-5v6b2" Jul 17 17:57:44.396979 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.396950 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 17 17:57:44.397119 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.397100 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-bmxnz\"" Jul 17 17:57:44.397353 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.397332 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 17 17:57:44.405142 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.404719 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-5v6b2"] Jul 17 17:57:44.460833 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.460718 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzzjp\" (UniqueName: \"kubernetes.io/projected/7a399b6a-5930-4576-9ec9-a1c6ccfc3a8f-kube-api-access-bzzjp\") pod \"downloads-5bc67dcd46-5v6b2\" (UID: \"7a399b6a-5930-4576-9ec9-a1c6ccfc3a8f\") " pod="openshift-console/downloads-5bc67dcd46-5v6b2" Jul 17 17:57:44.546624 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.546585 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfp68" event={"ID":"1f293570-245b-432e-b029-31aa06cd7156","Type":"ContainerStarted","Data":"ea5e2cdcd774b6ea4d737e34a010a0117b0f3a30c1c6617f6e88da7a259c16e5"} Jul 17 17:57:44.561166 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.561134 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzzjp\" (UniqueName: \"kubernetes.io/projected/7a399b6a-5930-4576-9ec9-a1c6ccfc3a8f-kube-api-access-bzzjp\") pod \"downloads-5bc67dcd46-5v6b2\" (UID: \"7a399b6a-5930-4576-9ec9-a1c6ccfc3a8f\") " pod="openshift-console/downloads-5bc67dcd46-5v6b2" Jul 17 17:57:44.561349 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.561249 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c7e5de6-b30e-4800-b952-332c75398e11-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.561418 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.561348 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.562452 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.562421 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c7e5de6-b30e-4800-b952-332c75398e11-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.564432 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.564385 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:44.574661 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.574630 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzzjp\" (UniqueName: \"kubernetes.io/projected/7a399b6a-5930-4576-9ec9-a1c6ccfc3a8f-kube-api-access-bzzjp\") pod \"downloads-5bc67dcd46-5v6b2\" (UID: \"7a399b6a-5930-4576-9ec9-a1c6ccfc3a8f\") " pod="openshift-console/downloads-5bc67dcd46-5v6b2" Jul 17 17:57:44.706267 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.706178 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-5v6b2" Jul 17 17:57:44.783683 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:44.783612 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:57:45.041222 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:45.041144 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:57:45.044101 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:57:45.044067 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7e5de6_b30e_4800_b952_332c75398e11.slice/crio-84654027fed58249a78230f68688e776463ddf63b9389345e6068e3775c0b994 WatchSource:0}: Error finding container 84654027fed58249a78230f68688e776463ddf63b9389345e6068e3775c0b994: Status 404 returned error can't find the container with id 84654027fed58249a78230f68688e776463ddf63b9389345e6068e3775c0b994 Jul 17 17:57:45.053924 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:45.053824 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-5v6b2"] Jul 17 17:57:45.058450 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:57:45.058429 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a399b6a_5930_4576_9ec9_a1c6ccfc3a8f.slice/crio-ca66f3e540bc538c5f16ed8d45fba28bc01cf19fe8dad5f7ac1953818702bb29 WatchSource:0}: Error finding container ca66f3e540bc538c5f16ed8d45fba28bc01cf19fe8dad5f7ac1953818702bb29: Status 404 returned error can't find the container with id ca66f3e540bc538c5f16ed8d45fba28bc01cf19fe8dad5f7ac1953818702bb29 Jul 17 17:57:45.552079 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:45.552038 2572 generic.go:356] "Generic (PLEG): container finished" podID="1f293570-245b-432e-b029-31aa06cd7156" containerID="3b4fd3cad8726bc0812677539a55e0a81ce646c917a2dfd9bde0f7fe73cb8228" exitCode=0 Jul 17 17:57:45.552265 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:45.552148 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfp68" event={"ID":"1f293570-245b-432e-b029-31aa06cd7156","Type":"ContainerDied","Data":"3b4fd3cad8726bc0812677539a55e0a81ce646c917a2dfd9bde0f7fe73cb8228"} Jul 17 17:57:45.553537 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:45.553501 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c7e5de6-b30e-4800-b952-332c75398e11","Type":"ContainerStarted","Data":"84654027fed58249a78230f68688e776463ddf63b9389345e6068e3775c0b994"} Jul 17 17:57:45.554685 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:45.554663 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-5v6b2" event={"ID":"7a399b6a-5930-4576-9ec9-a1c6ccfc3a8f","Type":"ContainerStarted","Data":"ca66f3e540bc538c5f16ed8d45fba28bc01cf19fe8dad5f7ac1953818702bb29"} Jul 17 17:57:47.201763 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.201674 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-84cfcc445d-gxlq7"] Jul 17 17:57:47.204044 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.204025 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.206748 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.206709 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 17 17:57:47.206982 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.206962 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 17 17:57:47.207279 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.207259 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7uk8q96nqulpq\"" Jul 17 17:57:47.207367 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.207283 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fwjq8\"" Jul 17 17:57:47.207367 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.207352 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 17 17:57:47.207956 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.207739 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 17 17:57:47.217396 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.217375 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84cfcc445d-gxlq7"] Jul 17 17:57:47.283531 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.283396 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a321161-6c33-48dd-8388-b39ed73c2379-secret-metrics-server-client-certs\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.283531 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.283457 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a321161-6c33-48dd-8388-b39ed73c2379-metrics-server-audit-profiles\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.283531 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.283487 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a321161-6c33-48dd-8388-b39ed73c2379-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.283825 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.283632 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a321161-6c33-48dd-8388-b39ed73c2379-secret-metrics-server-tls\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.283825 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.283728 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a321161-6c33-48dd-8388-b39ed73c2379-audit-log\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.283825 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.283801 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6p8g\" (UniqueName: \"kubernetes.io/projected/1a321161-6c33-48dd-8388-b39ed73c2379-kube-api-access-c6p8g\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.283977 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.283832 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a321161-6c33-48dd-8388-b39ed73c2379-client-ca-bundle\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.385456 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.385255 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a321161-6c33-48dd-8388-b39ed73c2379-secret-metrics-server-client-certs\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.385456 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.385315 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a321161-6c33-48dd-8388-b39ed73c2379-metrics-server-audit-profiles\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.385456 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.385347 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a321161-6c33-48dd-8388-b39ed73c2379-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.385456 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.385385 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a321161-6c33-48dd-8388-b39ed73c2379-secret-metrics-server-tls\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.386558 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.386504 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a321161-6c33-48dd-8388-b39ed73c2379-audit-log\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.386960 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.386921 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1a321161-6c33-48dd-8388-b39ed73c2379-audit-log\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.387386 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.387316 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1a321161-6c33-48dd-8388-b39ed73c2379-metrics-server-audit-profiles\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.387501 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.387391 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a321161-6c33-48dd-8388-b39ed73c2379-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.387581 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.387553 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6p8g\" (UniqueName: \"kubernetes.io/projected/1a321161-6c33-48dd-8388-b39ed73c2379-kube-api-access-c6p8g\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.387648 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.387604 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a321161-6c33-48dd-8388-b39ed73c2379-client-ca-bundle\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.391646 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.391621 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1a321161-6c33-48dd-8388-b39ed73c2379-secret-metrics-server-tls\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.392292 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.392255 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1a321161-6c33-48dd-8388-b39ed73c2379-secret-metrics-server-client-certs\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.394197 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.394175 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1a321161-6c33-48dd-8388-b39ed73c2379-client-ca-bundle\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.405066 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.405044 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6p8g\" (UniqueName: \"kubernetes.io/projected/1a321161-6c33-48dd-8388-b39ed73c2379-kube-api-access-c6p8g\") pod \"metrics-server-84cfcc445d-gxlq7\" (UID: \"1a321161-6c33-48dd-8388-b39ed73c2379\") " pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.462381 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.462341 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8v7vg" Jul 17 17:57:47.515899 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.515849 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:57:47.568359 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.568257 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g6q7x" event={"ID":"76665724-fb0c-43fe-b09d-e3d43ede2fc0","Type":"ContainerStarted","Data":"62459e67664b56aeb67227f547921df887c18b48d299802908d930e1d5dc5b59"} Jul 17 17:57:47.571265 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.571238 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfp68" event={"ID":"1f293570-245b-432e-b029-31aa06cd7156","Type":"ContainerStarted","Data":"4c2d580250bef56bc988ce900b6c0dad78fa21b13b7ddcf1775d26096ab4f4bb"} Jul 17 17:57:47.571387 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.571290 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfp68" event={"ID":"1f293570-245b-432e-b029-31aa06cd7156","Type":"ContainerStarted","Data":"5cb6c417c49c0c155e470a75d39d34a127e41c64c16cc3ac9cce9617e1b474e8"} Jul 17 17:57:47.591729 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.591661 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-g6q7x" podStartSLOduration=1.897372207 podStartE2EDuration="15.591643057s" totalImagesPullingTime="13.69427085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:57:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:47.590072175 +0000 UTC m=+62.999499808" watchObservedRunningTime="2026-07-17 17:57:47.591643057 +0000 UTC m=+63.001070707" Jul 17 17:57:47.615566 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.615479 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rfp68" podStartSLOduration=4.402160964 podStartE2EDuration="5.615463634s" totalImagesPullingTime="1.132458991s" totalInitContainerRuntime="80.843679ms" isStatefulPod=true podCreationTimestamp="2026-07-17 17:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:47.615217211 +0000 UTC m=+63.024644813" watchObservedRunningTime="2026-07-17 17:57:47.615463634 +0000 UTC m=+63.024891241" Jul 17 17:57:47.930604 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:47.930577 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-84cfcc445d-gxlq7"] Jul 17 17:57:47.963865 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:57:47.963827 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a321161_6c33_48dd_8388_b39ed73c2379.slice/crio-ee3109b1dc4a5a304559fac0e40683cc9de8b30a7a0349598fc04567c951f429 WatchSource:0}: Error finding container ee3109b1dc4a5a304559fac0e40683cc9de8b30a7a0349598fc04567c951f429: Status 404 returned error can't find the container with id ee3109b1dc4a5a304559fac0e40683cc9de8b30a7a0349598fc04567c951f429 Jul 17 17:57:48.027884 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.027850 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp"] Jul 17 17:57:48.030491 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.030471 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.033271 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.033246 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 17 17:57:48.033390 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.033351 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 17 17:57:48.038226 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.038023 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 17 17:57:48.038226 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.038056 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 17 17:57:48.038503 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.038418 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 17 17:57:48.038503 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.038440 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7bvp2\"" Jul 17 17:57:48.039202 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.038701 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 17 17:57:48.047838 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.047813 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp"] Jul 17 17:57:48.094242 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.094160 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e75a9a63-f13c-49d9-b156-4d45362e0cfe-secret-telemeter-client\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.094242 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.094206 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e75a9a63-f13c-49d9-b156-4d45362e0cfe-telemeter-client-tls\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.094242 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.094239 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj9sr\" (UniqueName: \"kubernetes.io/projected/e75a9a63-f13c-49d9-b156-4d45362e0cfe-kube-api-access-rj9sr\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.094519 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.094260 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e75a9a63-f13c-49d9-b156-4d45362e0cfe-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.094519 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.094293 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e75a9a63-f13c-49d9-b156-4d45362e0cfe-metrics-client-ca\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.094519 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.094410 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e75a9a63-f13c-49d9-b156-4d45362e0cfe-federate-client-tls\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.094519 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.094457 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e75a9a63-f13c-49d9-b156-4d45362e0cfe-serving-certs-ca-bundle\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.094667 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.094534 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e75a9a63-f13c-49d9-b156-4d45362e0cfe-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.195753 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.195720 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e75a9a63-f13c-49d9-b156-4d45362e0cfe-secret-telemeter-client\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.195753 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.195757 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e75a9a63-f13c-49d9-b156-4d45362e0cfe-telemeter-client-tls\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.196031 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.195890 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj9sr\" (UniqueName: \"kubernetes.io/projected/e75a9a63-f13c-49d9-b156-4d45362e0cfe-kube-api-access-rj9sr\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.196031 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.195938 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e75a9a63-f13c-49d9-b156-4d45362e0cfe-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.196031 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.195965 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e75a9a63-f13c-49d9-b156-4d45362e0cfe-metrics-client-ca\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.196185 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.196048 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e75a9a63-f13c-49d9-b156-4d45362e0cfe-federate-client-tls\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.196185 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.196087 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e75a9a63-f13c-49d9-b156-4d45362e0cfe-serving-certs-ca-bundle\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.196185 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.196134 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e75a9a63-f13c-49d9-b156-4d45362e0cfe-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.196913 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.196884 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e75a9a63-f13c-49d9-b156-4d45362e0cfe-metrics-client-ca\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.197076 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.196981 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e75a9a63-f13c-49d9-b156-4d45362e0cfe-serving-certs-ca-bundle\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.197311 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.197291 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e75a9a63-f13c-49d9-b156-4d45362e0cfe-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.198468 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.198441 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e75a9a63-f13c-49d9-b156-4d45362e0cfe-telemeter-client-tls\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.198693 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.198676 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e75a9a63-f13c-49d9-b156-4d45362e0cfe-secret-telemeter-client\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.198740 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.198724 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e75a9a63-f13c-49d9-b156-4d45362e0cfe-federate-client-tls\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.198827 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.198806 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e75a9a63-f13c-49d9-b156-4d45362e0cfe-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.204356 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.204336 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj9sr\" (UniqueName: \"kubernetes.io/projected/e75a9a63-f13c-49d9-b156-4d45362e0cfe-kube-api-access-rj9sr\") pod \"telemeter-client-79bc7d6f4f-pwjtp\" (UID: \"e75a9a63-f13c-49d9-b156-4d45362e0cfe\") " pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.342517 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.342484 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" Jul 17 17:57:48.489338 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.489301 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp"] Jul 17 17:57:48.492905 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:57:48.492879 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75a9a63_f13c_49d9_b156_4d45362e0cfe.slice/crio-ec4481783030b261179fd250e417b37136cc763faffdbce86082d02e55b59c94 WatchSource:0}: Error finding container ec4481783030b261179fd250e417b37136cc763faffdbce86082d02e55b59c94: Status 404 returned error can't find the container with id ec4481783030b261179fd250e417b37136cc763faffdbce86082d02e55b59c94 Jul 17 17:57:48.578378 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.578340 2572 generic.go:356] "Generic (PLEG): container finished" podID="9c7e5de6-b30e-4800-b952-332c75398e11" containerID="b2619d74f07d86240d73ded603d6222a59f9cbc644ba1cdedb946df217bb38e1" exitCode=0 Jul 17 17:57:48.578556 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.578444 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c7e5de6-b30e-4800-b952-332c75398e11","Type":"ContainerDied","Data":"b2619d74f07d86240d73ded603d6222a59f9cbc644ba1cdedb946df217bb38e1"} Jul 17 17:57:48.580950 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.580925 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" event={"ID":"e75a9a63-f13c-49d9-b156-4d45362e0cfe","Type":"ContainerStarted","Data":"ec4481783030b261179fd250e417b37136cc763faffdbce86082d02e55b59c94"} Jul 17 17:57:48.582602 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:48.582473 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" event={"ID":"1a321161-6c33-48dd-8388-b39ed73c2379","Type":"ContainerStarted","Data":"ee3109b1dc4a5a304559fac0e40683cc9de8b30a7a0349598fc04567c951f429"} Jul 17 17:57:49.035744 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.035707 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:57:49.039381 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.039360 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.042241 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.042219 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-44bm4sm79vu97\"" Jul 17 17:57:49.042359 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.042254 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dl25j\"" Jul 17 17:57:49.042546 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.042462 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 17 17:57:49.042679 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.042628 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 17 17:57:49.043189 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.043170 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 17 17:57:49.043417 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.043276 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 17 17:57:49.043527 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.043486 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 17 17:57:49.044128 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.043720 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 17 17:57:49.044128 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.043846 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 17 17:57:49.044128 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.043915 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 17 17:57:49.044128 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.043178 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 17 17:57:49.044128 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.044026 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 17 17:57:49.044128 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.044056 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 17 17:57:49.044462 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.044152 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 17 17:57:49.044462 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.044403 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 17 17:57:49.046243 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.046226 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 17 17:57:49.056074 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.056046 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:57:49.105073 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105025 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105233 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105080 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105233 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105111 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105233 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105143 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bdb2801-67f8-47c8-b0c4-449daf586d34-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105233 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105215 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bdb2801-67f8-47c8-b0c4-449daf586d34-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105504 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105254 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105504 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105294 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-config\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105504 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105350 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105504 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105405 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105504 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105447 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105504 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105484 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105945 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105509 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnmc8\" (UniqueName: \"kubernetes.io/projected/8bdb2801-67f8-47c8-b0c4-449daf586d34-kube-api-access-lnmc8\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105945 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105540 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105945 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105569 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105945 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105593 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105945 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105615 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105945 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105699 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105945 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105751 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105945 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105785 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.105945 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.105822 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.207104 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207063 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.207580 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207110 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.207580 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207142 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.207580 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207216 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.207580 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207262 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.207580 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207296 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.207580 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207327 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.207580 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207356 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.207580 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207382 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.207580 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207408 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.207580 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207443 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bdb2801-67f8-47c8-b0c4-449daf586d34-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.207580 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207484 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bdb2801-67f8-47c8-b0c4-449daf586d34-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.207580 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207508 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.207580 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207538 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-config\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.207580 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207566 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.208322 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207592 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.208322 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207617 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.208322 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207647 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.208322 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207671 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmc8\" (UniqueName: \"kubernetes.io/projected/8bdb2801-67f8-47c8-b0c4-449daf586d34-kube-api-access-lnmc8\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.208322 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.207702 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.209733 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.208917 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.209733 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.209089 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.209733 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.209398 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.209733 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.209694 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.210041 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.209963 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.210562 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.210247 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.214498 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.214191 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.214679 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.214647 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.215436 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.215362 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.216706 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.216299 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-web-config\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.219046 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.218021 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bdb2801-67f8-47c8-b0c4-449daf586d34-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.219046 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.218188 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bdb2801-67f8-47c8-b0c4-449daf586d34-config-out\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.219046 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.218582 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.219823 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.219274 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.219823 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.219426 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.219823 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.219764 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.221300 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.221276 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-config\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.221687 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.221668 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.222597 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.222537 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.223499 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.223460 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnmc8\" (UniqueName: \"kubernetes.io/projected/8bdb2801-67f8-47c8-b0c4-449daf586d34-kube-api-access-lnmc8\") pod \"prometheus-k8s-0\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.352317 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.352236 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:57:49.588414 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.588358 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" event={"ID":"1a321161-6c33-48dd-8388-b39ed73c2379","Type":"ContainerStarted","Data":"44c869e98adefb6ba6c3b8011f3d2a2eae7fbb6c060fc463adbb34ace83d49b1"} Jul 17 17:57:49.604906 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.604781 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:57:49.605512 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:49.605474 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" podStartSLOduration=1.103857362 podStartE2EDuration="2.605458157s" totalImagesPullingTime="1.501600795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:49.604847164 +0000 UTC m=+65.014274773" watchObservedRunningTime="2026-07-17 17:57:49.605458157 +0000 UTC m=+65.014885762" Jul 17 17:57:49.608129 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:57:49.608108 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bdb2801_67f8_47c8_b0c4_449daf586d34.slice/crio-5a969ac60e8c371356ea68a065f3a6a0446a0f5ba6eac6b4c1cd68f40bd948fa WatchSource:0}: Error finding container 5a969ac60e8c371356ea68a065f3a6a0446a0f5ba6eac6b4c1cd68f40bd948fa: Status 404 returned error can't find the container with id 5a969ac60e8c371356ea68a065f3a6a0446a0f5ba6eac6b4c1cd68f40bd948fa Jul 17 17:57:50.536048 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:50.535717 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nxcs2" Jul 17 17:57:50.597037 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:50.596899 2572 generic.go:356] "Generic (PLEG): container finished" podID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerID="136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605" exitCode=0 Jul 17 17:57:50.597037 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:50.596996 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdb2801-67f8-47c8-b0c4-449daf586d34","Type":"ContainerDied","Data":"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605"} Jul 17 17:57:50.597262 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:50.597101 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdb2801-67f8-47c8-b0c4-449daf586d34","Type":"ContainerStarted","Data":"5a969ac60e8c371356ea68a065f3a6a0446a0f5ba6eac6b4c1cd68f40bd948fa"} Jul 17 17:57:51.026486 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.026456 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltg2b\" (UniqueName: \"kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b\") pod \"network-check-target-wcf2q\" (UID: \"27188dfc-708b-462f-9a9a-24fda6997009\") " pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:51.026678 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.026500 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret\") pod \"global-pull-secret-syncer-ts454\" (UID: \"118b7937-fc4d-48fa-927e-2368a2317850\") " pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:51.026678 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.026539 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs\") pod \"network-metrics-daemon-5ccsk\" (UID: \"e05c7832-1be4-46bd-91e4-2dd5c8abeb02\") " pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:51.029378 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.029312 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 17:57:51.029378 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.029339 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 17:57:51.029378 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.029303 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 17:57:51.040361 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.040337 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/118b7937-fc4d-48fa-927e-2368a2317850-original-pull-secret\") pod \"global-pull-secret-syncer-ts454\" (UID: \"118b7937-fc4d-48fa-927e-2368a2317850\") " pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:51.040488 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.040435 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e05c7832-1be4-46bd-91e4-2dd5c8abeb02-metrics-certs\") pod \"network-metrics-daemon-5ccsk\" (UID: \"e05c7832-1be4-46bd-91e4-2dd5c8abeb02\") " pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:51.042082 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.042056 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 17:57:51.048715 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.048250 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ts454" Jul 17 17:57:51.051461 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.051440 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltg2b\" (UniqueName: \"kubernetes.io/projected/27188dfc-708b-462f-9a9a-24fda6997009-kube-api-access-ltg2b\") pod \"network-check-target-wcf2q\" (UID: \"27188dfc-708b-462f-9a9a-24fda6997009\") " pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:51.057562 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.057543 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jb7qg\"" Jul 17 17:57:51.065730 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.065710 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:57:51.337361 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.336173 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wcf2q"] Jul 17 17:57:51.341353 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.341332 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dn68c\"" Jul 17 17:57:51.342547 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:57:51.342522 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27188dfc_708b_462f_9a9a_24fda6997009.slice/crio-cc498c216479ce289ab1edd506721c22e2295730f888b4f468a6bd244d49162f WatchSource:0}: Error finding container cc498c216479ce289ab1edd506721c22e2295730f888b4f468a6bd244d49162f: Status 404 returned error can't find the container with id cc498c216479ce289ab1edd506721c22e2295730f888b4f468a6bd244d49162f Jul 17 17:57:51.349602 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.349465 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5ccsk" Jul 17 17:57:51.356154 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.356054 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ts454"] Jul 17 17:57:51.361144 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:57:51.360799 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118b7937_fc4d_48fa_927e_2368a2317850.slice/crio-544e6765aff95a77a927a898398b2d971e8be2f2bc4488d687ef954ae5d4a758 WatchSource:0}: Error finding container 544e6765aff95a77a927a898398b2d971e8be2f2bc4488d687ef954ae5d4a758: Status 404 returned error can't find the container with id 544e6765aff95a77a927a898398b2d971e8be2f2bc4488d687ef954ae5d4a758 Jul 17 17:57:51.527234 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.526480 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5ccsk"] Jul 17 17:57:51.530931 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:57:51.530889 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05c7832_1be4_46bd_91e4_2dd5c8abeb02.slice/crio-89fb8e6cbdf1eeaefc750bbab57184024c7e318a7ac806faa10389c83a7150db WatchSource:0}: Error finding container 89fb8e6cbdf1eeaefc750bbab57184024c7e318a7ac806faa10389c83a7150db: Status 404 returned error can't find the container with id 89fb8e6cbdf1eeaefc750bbab57184024c7e318a7ac806faa10389c83a7150db Jul 17 17:57:51.607372 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.607337 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c7e5de6-b30e-4800-b952-332c75398e11","Type":"ContainerStarted","Data":"b67f6369b86d6cdef308be834850fc092748f80bb679c68cb100322e304ea29c"} Jul 17 17:57:51.607776 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.607414 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c7e5de6-b30e-4800-b952-332c75398e11","Type":"ContainerStarted","Data":"66c4827d251c39a3116b351b592122bd6a27e7c0d5771276d5086e998d752354"} Jul 17 17:57:51.607776 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.607458 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c7e5de6-b30e-4800-b952-332c75398e11","Type":"ContainerStarted","Data":"5b52ef622593aeadfd94eb3f386ec6fd4e2ed7e700f72d9709eec00b8208101c"} Jul 17 17:57:51.607776 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.607500 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c7e5de6-b30e-4800-b952-332c75398e11","Type":"ContainerStarted","Data":"083da51f9882f67ea4fe9a665b2b268b2bf5b829c860d0b317ac6eb0949524c4"} Jul 17 17:57:51.607776 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.607535 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c7e5de6-b30e-4800-b952-332c75398e11","Type":"ContainerStarted","Data":"f8370e691290c045f38cbb65e10065cf294169eadd9f72a87ff262bc6f3405da"} Jul 17 17:57:51.609370 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.609341 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wcf2q" event={"ID":"27188dfc-708b-462f-9a9a-24fda6997009","Type":"ContainerStarted","Data":"cc498c216479ce289ab1edd506721c22e2295730f888b4f468a6bd244d49162f"} Jul 17 17:57:51.613611 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.613290 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" event={"ID":"e75a9a63-f13c-49d9-b156-4d45362e0cfe","Type":"ContainerStarted","Data":"c4654452f35d5d351a5cd78a38dd0181323b55c52c71ef83ec16963eee47afa2"} Jul 17 17:57:51.613611 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.613343 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" event={"ID":"e75a9a63-f13c-49d9-b156-4d45362e0cfe","Type":"ContainerStarted","Data":"a06a5f6bd9c25a0bbd8fe2749a8894e9b47b14aa2268861d7ba23eee05e4f634"} Jul 17 17:57:51.613611 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.613371 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" event={"ID":"e75a9a63-f13c-49d9-b156-4d45362e0cfe","Type":"ContainerStarted","Data":"1983ffc3d56464164c0b8ad2ec09c05a98412b9b9a5dfd3b2b90b4e27dc327e6"} Jul 17 17:57:51.614878 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.614848 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5ccsk" event={"ID":"e05c7832-1be4-46bd-91e4-2dd5c8abeb02","Type":"ContainerStarted","Data":"89fb8e6cbdf1eeaefc750bbab57184024c7e318a7ac806faa10389c83a7150db"} Jul 17 17:57:51.616091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.616036 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ts454" event={"ID":"118b7937-fc4d-48fa-927e-2368a2317850","Type":"ContainerStarted","Data":"544e6765aff95a77a927a898398b2d971e8be2f2bc4488d687ef954ae5d4a758"} Jul 17 17:57:51.634446 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:51.634330 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-79bc7d6f4f-pwjtp" podStartSLOduration=0.944163005 podStartE2EDuration="3.634310418s" totalImagesPullingTime="2.690147413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:51.63351869 +0000 UTC m=+67.042946299" watchObservedRunningTime="2026-07-17 17:57:51.634310418 +0000 UTC m=+67.043738024" Jul 17 17:57:53.636052 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:53.635896 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c7e5de6-b30e-4800-b952-332c75398e11","Type":"ContainerStarted","Data":"d1ce6e8a59979f3e8793f9d61ae6aca466964f22164104d750d51fc4b251d186"} Jul 17 17:57:53.640588 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:53.640555 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5ccsk" event={"ID":"e05c7832-1be4-46bd-91e4-2dd5c8abeb02","Type":"ContainerStarted","Data":"47d29ebd9398a427f570d79190590407f3b6a62b5f4966820d23d849e4da4ed6"} Jul 17 17:57:53.640718 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:53.640614 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5ccsk" event={"ID":"e05c7832-1be4-46bd-91e4-2dd5c8abeb02","Type":"ContainerStarted","Data":"c472a9e7a9b204b4d90872bc4e104bb88c32629223caf07e8b4fb69635f4ca18"} Jul 17 17:57:53.667255 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:53.666566 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.8290205029999997 podStartE2EDuration="10.666545081s" totalImagesPullingTime="6.821531035s" totalInitContainerRuntime="15.993543ms" isStatefulPod=false podCreationTimestamp="2026-07-17 17:57:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:53.662825905 +0000 UTC m=+69.072253526" watchObservedRunningTime="2026-07-17 17:57:53.666545081 +0000 UTC m=+69.075972688" Jul 17 17:57:53.681743 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:53.680702 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5ccsk" podStartSLOduration=66.179597128 podStartE2EDuration="1m7.680665788s" totalImagesPullingTime="1.50106866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:53.679619887 +0000 UTC m=+69.089047494" watchObservedRunningTime="2026-07-17 17:57:53.680665788 +0000 UTC m=+69.090093396" Jul 17 17:57:54.513066 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:57:54.513034 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b558bd496-vjjrw" Jul 17 17:58:05.694747 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:05.694715 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdb2801-67f8-47c8-b0c4-449daf586d34","Type":"ContainerStarted","Data":"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177"} Jul 17 17:58:05.697087 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:05.697057 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wcf2q" event={"ID":"27188dfc-708b-462f-9a9a-24fda6997009","Type":"ContainerStarted","Data":"e33cb047fb349e3ff9c63533d309baca1a524b8f517b5c06762f47d137980182"} Jul 17 17:58:05.697259 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:05.697241 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:58:05.712749 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:05.712703 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wcf2q" podStartSLOduration=66.519156431 podStartE2EDuration="1m20.712685552s" totalImagesPullingTime="14.193529121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:58:05.710975901 +0000 UTC m=+81.120403528" watchObservedRunningTime="2026-07-17 17:58:05.712685552 +0000 UTC m=+81.122113159" Jul 17 17:58:06.705670 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:06.705621 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdb2801-67f8-47c8-b0c4-449daf586d34","Type":"ContainerStarted","Data":"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf"} Jul 17 17:58:06.707979 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:06.707945 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ts454" event={"ID":"118b7937-fc4d-48fa-927e-2368a2317850","Type":"ContainerStarted","Data":"50646b2ab05ee41cada94d9ca2469eba862aea4826fb905a87c89961373b57db"} Jul 17 17:58:06.711815 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:06.711603 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-5v6b2" event={"ID":"7a399b6a-5930-4576-9ec9-a1c6ccfc3a8f","Type":"ContainerStarted","Data":"0a8af9f14dc1eef2ce94e9cb6b4e82486ec7d9014a8c1669641f00d4ebda35f2"} Jul 17 17:58:06.712143 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:06.712070 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-5v6b2" Jul 17 17:58:06.722273 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:06.722251 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-5v6b2" Jul 17 17:58:06.741131 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:06.741079 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ts454" podStartSLOduration=66.552681479 podStartE2EDuration="1m20.741058328s" totalImagesPullingTime="14.188376849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:58:06.724848074 +0000 UTC m=+82.134275680" watchObservedRunningTime="2026-07-17 17:58:06.741058328 +0000 UTC m=+82.150485935" Jul 17 17:58:06.742230 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:06.741761 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-5v6b2" podStartSLOduration=1.908767528 podStartE2EDuration="22.74174867s" totalImagesPullingTime="20.832981142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:58:06.740301998 +0000 UTC m=+82.149729606" watchObservedRunningTime="2026-07-17 17:58:06.74174867 +0000 UTC m=+82.151176279" Jul 17 17:58:07.516137 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:07.516098 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:58:07.516329 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:07.516174 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:58:08.725758 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:08.725717 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdb2801-67f8-47c8-b0c4-449daf586d34","Type":"ContainerStarted","Data":"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda"} Jul 17 17:58:09.734078 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:09.734039 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdb2801-67f8-47c8-b0c4-449daf586d34","Type":"ContainerStarted","Data":"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983"} Jul 17 17:58:09.734540 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:09.734108 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdb2801-67f8-47c8-b0c4-449daf586d34","Type":"ContainerStarted","Data":"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b"} Jul 17 17:58:09.734540 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:09.734137 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdb2801-67f8-47c8-b0c4-449daf586d34","Type":"ContainerStarted","Data":"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc"} Jul 17 17:58:09.766584 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:09.766517 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.27134768 podStartE2EDuration="20.766496402s" totalImagesPullingTime="17.46763243s" totalInitContainerRuntime="27.516292ms" isStatefulPod=false podCreationTimestamp="2026-07-17 17:57:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:58:09.764495294 +0000 UTC m=+85.173922944" watchObservedRunningTime="2026-07-17 17:58:09.766496402 +0000 UTC m=+85.175924009" Jul 17 17:58:14.352706 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:14.352669 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:58:27.521893 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:27.521859 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:58:27.525835 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:27.525813 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-84cfcc445d-gxlq7" Jul 17 17:58:36.714348 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:36.714321 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wcf2q" Jul 17 17:58:49.352882 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:49.352840 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:58:49.416298 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:49.416270 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:58:49.904396 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:58:49.904370 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:03.119565 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:03.119530 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:59:03.120158 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:03.119886 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="alertmanager" containerID="cri-o://f8370e691290c045f38cbb65e10065cf294169eadd9f72a87ff262bc6f3405da" gracePeriod=120 Jul 17 17:59:03.120158 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:03.119953 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="kube-rbac-proxy-metric" containerID="cri-o://b67f6369b86d6cdef308be834850fc092748f80bb679c68cb100322e304ea29c" gracePeriod=120 Jul 17 17:59:03.120158 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:03.119973 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="kube-rbac-proxy-web" containerID="cri-o://5b52ef622593aeadfd94eb3f386ec6fd4e2ed7e700f72d9709eec00b8208101c" gracePeriod=120 Jul 17 17:59:03.120158 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:03.120024 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="config-reloader" containerID="cri-o://083da51f9882f67ea4fe9a665b2b268b2bf5b829c860d0b317ac6eb0949524c4" gracePeriod=120 Jul 17 17:59:03.120158 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:03.119985 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="kube-rbac-proxy" containerID="cri-o://66c4827d251c39a3116b351b592122bd6a27e7c0d5771276d5086e998d752354" gracePeriod=120 Jul 17 17:59:03.120158 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:03.120078 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="prom-label-proxy" containerID="cri-o://d1ce6e8a59979f3e8793f9d61ae6aca466964f22164104d750d51fc4b251d186" gracePeriod=120 Jul 17 17:59:03.943229 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:03.943199 2572 generic.go:356] "Generic (PLEG): container finished" podID="9c7e5de6-b30e-4800-b952-332c75398e11" containerID="d1ce6e8a59979f3e8793f9d61ae6aca466964f22164104d750d51fc4b251d186" exitCode=0 Jul 17 17:59:03.943229 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:03.943222 2572 generic.go:356] "Generic (PLEG): container finished" podID="9c7e5de6-b30e-4800-b952-332c75398e11" containerID="b67f6369b86d6cdef308be834850fc092748f80bb679c68cb100322e304ea29c" exitCode=0 Jul 17 17:59:03.943229 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:03.943228 2572 generic.go:356] "Generic (PLEG): container finished" podID="9c7e5de6-b30e-4800-b952-332c75398e11" containerID="66c4827d251c39a3116b351b592122bd6a27e7c0d5771276d5086e998d752354" exitCode=0 Jul 17 17:59:03.943229 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:03.943234 2572 generic.go:356] "Generic (PLEG): container finished" podID="9c7e5de6-b30e-4800-b952-332c75398e11" containerID="083da51f9882f67ea4fe9a665b2b268b2bf5b829c860d0b317ac6eb0949524c4" exitCode=0 Jul 17 17:59:03.943229 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:03.943239 2572 generic.go:356] "Generic (PLEG): container finished" podID="9c7e5de6-b30e-4800-b952-332c75398e11" containerID="f8370e691290c045f38cbb65e10065cf294169eadd9f72a87ff262bc6f3405da" exitCode=0 Jul 17 17:59:03.943593 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:03.943278 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c7e5de6-b30e-4800-b952-332c75398e11","Type":"ContainerDied","Data":"d1ce6e8a59979f3e8793f9d61ae6aca466964f22164104d750d51fc4b251d186"} Jul 17 17:59:03.943593 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:03.943358 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c7e5de6-b30e-4800-b952-332c75398e11","Type":"ContainerDied","Data":"b67f6369b86d6cdef308be834850fc092748f80bb679c68cb100322e304ea29c"} Jul 17 17:59:03.943593 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:03.943400 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c7e5de6-b30e-4800-b952-332c75398e11","Type":"ContainerDied","Data":"66c4827d251c39a3116b351b592122bd6a27e7c0d5771276d5086e998d752354"} Jul 17 17:59:03.943593 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:03.943435 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c7e5de6-b30e-4800-b952-332c75398e11","Type":"ContainerDied","Data":"083da51f9882f67ea4fe9a665b2b268b2bf5b829c860d0b317ac6eb0949524c4"} Jul 17 17:59:03.943593 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:03.943477 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c7e5de6-b30e-4800-b952-332c75398e11","Type":"ContainerDied","Data":"f8370e691290c045f38cbb65e10065cf294169eadd9f72a87ff262bc6f3405da"} Jul 17 17:59:04.355904 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.355881 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:59:04.474816 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.474785 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c7e5de6-b30e-4800-b952-332c75398e11-alertmanager-main-db\") pod \"9c7e5de6-b30e-4800-b952-332c75398e11\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " Jul 17 17:59:04.474976 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.474835 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-main-tls\") pod \"9c7e5de6-b30e-4800-b952-332c75398e11\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " Jul 17 17:59:04.474976 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.474858 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-kube-rbac-proxy\") pod \"9c7e5de6-b30e-4800-b952-332c75398e11\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " Jul 17 17:59:04.474976 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.474885 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c7e5de6-b30e-4800-b952-332c75398e11-config-out\") pod \"9c7e5de6-b30e-4800-b952-332c75398e11\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " Jul 17 17:59:04.474976 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.474915 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-md5jt\" (UniqueName: \"kubernetes.io/projected/9c7e5de6-b30e-4800-b952-332c75398e11-kube-api-access-md5jt\") pod \"9c7e5de6-b30e-4800-b952-332c75398e11\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " Jul 17 17:59:04.474976 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.474936 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c7e5de6-b30e-4800-b952-332c75398e11-tls-assets\") pod \"9c7e5de6-b30e-4800-b952-332c75398e11\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " Jul 17 17:59:04.475259 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.474978 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c7e5de6-b30e-4800-b952-332c75398e11-alertmanager-trusted-ca-bundle\") pod \"9c7e5de6-b30e-4800-b952-332c75398e11\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " Jul 17 17:59:04.475259 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.475004 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-cluster-tls-config\") pod \"9c7e5de6-b30e-4800-b952-332c75398e11\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " Jul 17 17:59:04.475259 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.475067 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9c7e5de6-b30e-4800-b952-332c75398e11\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " Jul 17 17:59:04.475259 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.475131 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-kube-rbac-proxy-web\") pod \"9c7e5de6-b30e-4800-b952-332c75398e11\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " Jul 17 17:59:04.475259 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.475156 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-config-volume\") pod \"9c7e5de6-b30e-4800-b952-332c75398e11\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " Jul 17 17:59:04.475259 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.475181 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-web-config\") pod \"9c7e5de6-b30e-4800-b952-332c75398e11\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " Jul 17 17:59:04.475259 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.475175 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c7e5de6-b30e-4800-b952-332c75398e11-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "9c7e5de6-b30e-4800-b952-332c75398e11" (UID: "9c7e5de6-b30e-4800-b952-332c75398e11"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:59:04.475259 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.475226 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c7e5de6-b30e-4800-b952-332c75398e11-metrics-client-ca\") pod \"9c7e5de6-b30e-4800-b952-332c75398e11\" (UID: \"9c7e5de6-b30e-4800-b952-332c75398e11\") " Jul 17 17:59:04.475614 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.475479 2572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c7e5de6-b30e-4800-b952-332c75398e11-alertmanager-main-db\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:04.475932 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.475866 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7e5de6-b30e-4800-b952-332c75398e11-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "9c7e5de6-b30e-4800-b952-332c75398e11" (UID: "9c7e5de6-b30e-4800-b952-332c75398e11"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:59:04.476065 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.476049 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7e5de6-b30e-4800-b952-332c75398e11-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "9c7e5de6-b30e-4800-b952-332c75398e11" (UID: "9c7e5de6-b30e-4800-b952-332c75398e11"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:59:04.478198 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.478118 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c7e5de6-b30e-4800-b952-332c75398e11-config-out" (OuterVolumeSpecName: "config-out") pod "9c7e5de6-b30e-4800-b952-332c75398e11" (UID: "9c7e5de6-b30e-4800-b952-332c75398e11"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:59:04.478832 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.478743 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "9c7e5de6-b30e-4800-b952-332c75398e11" (UID: "9c7e5de6-b30e-4800-b952-332c75398e11"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:59:04.478832 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.478744 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7e5de6-b30e-4800-b952-332c75398e11-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9c7e5de6-b30e-4800-b952-332c75398e11" (UID: "9c7e5de6-b30e-4800-b952-332c75398e11"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:59:04.478972 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.478949 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7e5de6-b30e-4800-b952-332c75398e11-kube-api-access-md5jt" (OuterVolumeSpecName: "kube-api-access-md5jt") pod "9c7e5de6-b30e-4800-b952-332c75398e11" (UID: "9c7e5de6-b30e-4800-b952-332c75398e11"). InnerVolumeSpecName "kube-api-access-md5jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:59:04.479517 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.479491 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "9c7e5de6-b30e-4800-b952-332c75398e11" (UID: "9c7e5de6-b30e-4800-b952-332c75398e11"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:59:04.479601 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.479546 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "9c7e5de6-b30e-4800-b952-332c75398e11" (UID: "9c7e5de6-b30e-4800-b952-332c75398e11"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:59:04.479663 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.479635 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "9c7e5de6-b30e-4800-b952-332c75398e11" (UID: "9c7e5de6-b30e-4800-b952-332c75398e11"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:59:04.479751 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.479729 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-config-volume" (OuterVolumeSpecName: "config-volume") pod "9c7e5de6-b30e-4800-b952-332c75398e11" (UID: "9c7e5de6-b30e-4800-b952-332c75398e11"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:59:04.482439 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.482322 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "9c7e5de6-b30e-4800-b952-332c75398e11" (UID: "9c7e5de6-b30e-4800-b952-332c75398e11"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:59:04.488282 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.488234 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-web-config" (OuterVolumeSpecName: "web-config") pod "9c7e5de6-b30e-4800-b952-332c75398e11" (UID: "9c7e5de6-b30e-4800-b952-332c75398e11"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:59:04.576691 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.576660 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c7e5de6-b30e-4800-b952-332c75398e11-metrics-client-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:04.576691 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.576686 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-main-tls\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:04.576851 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.576698 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:04.576851 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.576708 2572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c7e5de6-b30e-4800-b952-332c75398e11-config-out\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:04.576851 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.576717 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md5jt\" (UniqueName: \"kubernetes.io/projected/9c7e5de6-b30e-4800-b952-332c75398e11-kube-api-access-md5jt\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:04.576851 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.576725 2572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c7e5de6-b30e-4800-b952-332c75398e11-tls-assets\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:04.576851 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.576735 2572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c7e5de6-b30e-4800-b952-332c75398e11-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:04.576851 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.576744 2572 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-cluster-tls-config\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:04.576851 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.576752 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:04.576851 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.576762 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:04.576851 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.576770 2572 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-config-volume\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:04.576851 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.576779 2572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c7e5de6-b30e-4800-b952-332c75398e11-web-config\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:04.950052 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.949997 2572 generic.go:356] "Generic (PLEG): container finished" podID="9c7e5de6-b30e-4800-b952-332c75398e11" containerID="5b52ef622593aeadfd94eb3f386ec6fd4e2ed7e700f72d9709eec00b8208101c" exitCode=0 Jul 17 17:59:04.950052 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.950041 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c7e5de6-b30e-4800-b952-332c75398e11","Type":"ContainerDied","Data":"5b52ef622593aeadfd94eb3f386ec6fd4e2ed7e700f72d9709eec00b8208101c"} Jul 17 17:59:04.950260 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.950107 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:59:04.950260 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.950133 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c7e5de6-b30e-4800-b952-332c75398e11","Type":"ContainerDied","Data":"84654027fed58249a78230f68688e776463ddf63b9389345e6068e3775c0b994"} Jul 17 17:59:04.950260 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.950168 2572 scope.go:117] "RemoveContainer" containerID="d1ce6e8a59979f3e8793f9d61ae6aca466964f22164104d750d51fc4b251d186" Jul 17 17:59:04.958440 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.958426 2572 scope.go:117] "RemoveContainer" containerID="b67f6369b86d6cdef308be834850fc092748f80bb679c68cb100322e304ea29c" Jul 17 17:59:04.964730 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.964715 2572 scope.go:117] "RemoveContainer" containerID="66c4827d251c39a3116b351b592122bd6a27e7c0d5771276d5086e998d752354" Jul 17 17:59:04.971078 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.971055 2572 scope.go:117] "RemoveContainer" containerID="5b52ef622593aeadfd94eb3f386ec6fd4e2ed7e700f72d9709eec00b8208101c" Jul 17 17:59:04.974135 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.974115 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:59:04.977650 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.977621 2572 scope.go:117] "RemoveContainer" containerID="083da51f9882f67ea4fe9a665b2b268b2bf5b829c860d0b317ac6eb0949524c4" Jul 17 17:59:04.982408 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.982390 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:59:04.984699 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.984685 2572 scope.go:117] "RemoveContainer" containerID="f8370e691290c045f38cbb65e10065cf294169eadd9f72a87ff262bc6f3405da" Jul 17 17:59:04.991113 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.991092 2572 scope.go:117] "RemoveContainer" containerID="b2619d74f07d86240d73ded603d6222a59f9cbc644ba1cdedb946df217bb38e1" Jul 17 17:59:04.997432 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.997406 2572 scope.go:117] "RemoveContainer" containerID="d1ce6e8a59979f3e8793f9d61ae6aca466964f22164104d750d51fc4b251d186" Jul 17 17:59:04.997677 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:59:04.997657 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ce6e8a59979f3e8793f9d61ae6aca466964f22164104d750d51fc4b251d186\": container with ID starting with d1ce6e8a59979f3e8793f9d61ae6aca466964f22164104d750d51fc4b251d186 not found: ID does not exist" containerID="d1ce6e8a59979f3e8793f9d61ae6aca466964f22164104d750d51fc4b251d186" Jul 17 17:59:04.997726 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.997686 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ce6e8a59979f3e8793f9d61ae6aca466964f22164104d750d51fc4b251d186"} err="failed to get container status \"d1ce6e8a59979f3e8793f9d61ae6aca466964f22164104d750d51fc4b251d186\": rpc error: code = NotFound desc = could not find container \"d1ce6e8a59979f3e8793f9d61ae6aca466964f22164104d750d51fc4b251d186\": container with ID starting with d1ce6e8a59979f3e8793f9d61ae6aca466964f22164104d750d51fc4b251d186 not found: ID does not exist" Jul 17 17:59:04.997773 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.997725 2572 scope.go:117] "RemoveContainer" containerID="b67f6369b86d6cdef308be834850fc092748f80bb679c68cb100322e304ea29c" Jul 17 17:59:04.997960 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:59:04.997943 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b67f6369b86d6cdef308be834850fc092748f80bb679c68cb100322e304ea29c\": container with ID starting with b67f6369b86d6cdef308be834850fc092748f80bb679c68cb100322e304ea29c not found: ID does not exist" containerID="b67f6369b86d6cdef308be834850fc092748f80bb679c68cb100322e304ea29c" Jul 17 17:59:04.998035 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.997965 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b67f6369b86d6cdef308be834850fc092748f80bb679c68cb100322e304ea29c"} err="failed to get container status \"b67f6369b86d6cdef308be834850fc092748f80bb679c68cb100322e304ea29c\": rpc error: code = NotFound desc = could not find container \"b67f6369b86d6cdef308be834850fc092748f80bb679c68cb100322e304ea29c\": container with ID starting with b67f6369b86d6cdef308be834850fc092748f80bb679c68cb100322e304ea29c not found: ID does not exist" Jul 17 17:59:04.998035 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.997981 2572 scope.go:117] "RemoveContainer" containerID="66c4827d251c39a3116b351b592122bd6a27e7c0d5771276d5086e998d752354" Jul 17 17:59:04.998289 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:59:04.998273 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c4827d251c39a3116b351b592122bd6a27e7c0d5771276d5086e998d752354\": container with ID starting with 66c4827d251c39a3116b351b592122bd6a27e7c0d5771276d5086e998d752354 not found: ID does not exist" containerID="66c4827d251c39a3116b351b592122bd6a27e7c0d5771276d5086e998d752354" Jul 17 17:59:04.998342 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.998294 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c4827d251c39a3116b351b592122bd6a27e7c0d5771276d5086e998d752354"} err="failed to get container status \"66c4827d251c39a3116b351b592122bd6a27e7c0d5771276d5086e998d752354\": rpc error: code = NotFound desc = could not find container \"66c4827d251c39a3116b351b592122bd6a27e7c0d5771276d5086e998d752354\": container with ID starting with 66c4827d251c39a3116b351b592122bd6a27e7c0d5771276d5086e998d752354 not found: ID does not exist" Jul 17 17:59:04.998342 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.998309 2572 scope.go:117] "RemoveContainer" containerID="5b52ef622593aeadfd94eb3f386ec6fd4e2ed7e700f72d9709eec00b8208101c" Jul 17 17:59:04.998521 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:59:04.998502 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b52ef622593aeadfd94eb3f386ec6fd4e2ed7e700f72d9709eec00b8208101c\": container with ID starting with 5b52ef622593aeadfd94eb3f386ec6fd4e2ed7e700f72d9709eec00b8208101c not found: ID does not exist" containerID="5b52ef622593aeadfd94eb3f386ec6fd4e2ed7e700f72d9709eec00b8208101c" Jul 17 17:59:04.998579 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.998529 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b52ef622593aeadfd94eb3f386ec6fd4e2ed7e700f72d9709eec00b8208101c"} err="failed to get container status \"5b52ef622593aeadfd94eb3f386ec6fd4e2ed7e700f72d9709eec00b8208101c\": rpc error: code = NotFound desc = could not find container \"5b52ef622593aeadfd94eb3f386ec6fd4e2ed7e700f72d9709eec00b8208101c\": container with ID starting with 5b52ef622593aeadfd94eb3f386ec6fd4e2ed7e700f72d9709eec00b8208101c not found: ID does not exist" Jul 17 17:59:04.998579 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.998552 2572 scope.go:117] "RemoveContainer" containerID="083da51f9882f67ea4fe9a665b2b268b2bf5b829c860d0b317ac6eb0949524c4" Jul 17 17:59:04.998780 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:59:04.998764 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"083da51f9882f67ea4fe9a665b2b268b2bf5b829c860d0b317ac6eb0949524c4\": container with ID starting with 083da51f9882f67ea4fe9a665b2b268b2bf5b829c860d0b317ac6eb0949524c4 not found: ID does not exist" containerID="083da51f9882f67ea4fe9a665b2b268b2bf5b829c860d0b317ac6eb0949524c4" Jul 17 17:59:04.998822 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.998785 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"083da51f9882f67ea4fe9a665b2b268b2bf5b829c860d0b317ac6eb0949524c4"} err="failed to get container status \"083da51f9882f67ea4fe9a665b2b268b2bf5b829c860d0b317ac6eb0949524c4\": rpc error: code = NotFound desc = could not find container \"083da51f9882f67ea4fe9a665b2b268b2bf5b829c860d0b317ac6eb0949524c4\": container with ID starting with 083da51f9882f67ea4fe9a665b2b268b2bf5b829c860d0b317ac6eb0949524c4 not found: ID does not exist" Jul 17 17:59:04.998822 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.998799 2572 scope.go:117] "RemoveContainer" containerID="f8370e691290c045f38cbb65e10065cf294169eadd9f72a87ff262bc6f3405da" Jul 17 17:59:04.998994 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:59:04.998979 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8370e691290c045f38cbb65e10065cf294169eadd9f72a87ff262bc6f3405da\": container with ID starting with f8370e691290c045f38cbb65e10065cf294169eadd9f72a87ff262bc6f3405da not found: ID does not exist" containerID="f8370e691290c045f38cbb65e10065cf294169eadd9f72a87ff262bc6f3405da" Jul 17 17:59:04.999065 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.998996 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8370e691290c045f38cbb65e10065cf294169eadd9f72a87ff262bc6f3405da"} err="failed to get container status \"f8370e691290c045f38cbb65e10065cf294169eadd9f72a87ff262bc6f3405da\": rpc error: code = NotFound desc = could not find container \"f8370e691290c045f38cbb65e10065cf294169eadd9f72a87ff262bc6f3405da\": container with ID starting with f8370e691290c045f38cbb65e10065cf294169eadd9f72a87ff262bc6f3405da not found: ID does not exist" Jul 17 17:59:04.999065 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.999023 2572 scope.go:117] "RemoveContainer" containerID="b2619d74f07d86240d73ded603d6222a59f9cbc644ba1cdedb946df217bb38e1" Jul 17 17:59:04.999277 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:59:04.999250 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2619d74f07d86240d73ded603d6222a59f9cbc644ba1cdedb946df217bb38e1\": container with ID starting with b2619d74f07d86240d73ded603d6222a59f9cbc644ba1cdedb946df217bb38e1 not found: ID does not exist" containerID="b2619d74f07d86240d73ded603d6222a59f9cbc644ba1cdedb946df217bb38e1" Jul 17 17:59:04.999327 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:04.999285 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2619d74f07d86240d73ded603d6222a59f9cbc644ba1cdedb946df217bb38e1"} err="failed to get container status \"b2619d74f07d86240d73ded603d6222a59f9cbc644ba1cdedb946df217bb38e1\": rpc error: code = NotFound desc = could not find container \"b2619d74f07d86240d73ded603d6222a59f9cbc644ba1cdedb946df217bb38e1\": container with ID starting with b2619d74f07d86240d73ded603d6222a59f9cbc644ba1cdedb946df217bb38e1 not found: ID does not exist" Jul 17 17:59:05.223959 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:05.223892 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" path="/var/lib/kubelet/pods/9c7e5de6-b30e-4800-b952-332c75398e11/volumes" Jul 17 17:59:07.344389 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.344349 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:59:07.344856 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.344710 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="prometheus" containerID="cri-o://e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177" gracePeriod=600 Jul 17 17:59:07.344856 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.344725 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="kube-rbac-proxy" containerID="cri-o://8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b" gracePeriod=600 Jul 17 17:59:07.344856 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.344754 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="kube-rbac-proxy-web" containerID="cri-o://39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc" gracePeriod=600 Jul 17 17:59:07.344856 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.344760 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="config-reloader" containerID="cri-o://d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf" gracePeriod=600 Jul 17 17:59:07.344856 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.344776 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983" gracePeriod=600 Jul 17 17:59:07.344856 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.344728 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="thanos-sidecar" containerID="cri-o://88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda" gracePeriod=600 Jul 17 17:59:07.580928 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.580896 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:07.596700 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.596653 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-configmap-serving-certs-ca-bundle\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.596700 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.596681 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-rulefiles-2\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.596875 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.596707 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.596875 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.596732 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-configmap-metrics-client-ca\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.596875 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.596758 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-configmap-kubelet-serving-ca-bundle\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.596875 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.596777 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-rulefiles-1\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.596875 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.596798 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.596875 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.596817 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-config\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.596875 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.596842 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-metrics-client-certs\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.596875 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.596875 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-trusted-ca-bundle\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.597284 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.596900 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-rulefiles-0\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.597284 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.596922 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-grpc-tls\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.597284 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.596951 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bdb2801-67f8-47c8-b0c4-449daf586d34-tls-assets\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.597284 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.596975 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-thanos-prometheus-http-client-file\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.597284 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.597033 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnmc8\" (UniqueName: \"kubernetes.io/projected/8bdb2801-67f8-47c8-b0c4-449daf586d34-kube-api-access-lnmc8\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.597284 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.597077 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-kube-rbac-proxy\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.597284 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.597113 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:59:07.597284 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.597124 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-db\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.597284 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.597141 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:59:07.597284 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.597186 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bdb2801-67f8-47c8-b0c4-449daf586d34-config-out\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.597769 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.597332 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:59:07.597769 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.597266 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:59:07.597769 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.597450 2572 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.597769 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.597469 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.597769 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.597484 2572 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-configmap-metrics-client-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.597769 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.597499 2572 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.598098 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.597778 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:59:07.599517 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.599489 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:59:07.600561 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.600257 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:59:07.602039 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.600800 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bdb2801-67f8-47c8-b0c4-449daf586d34-kube-api-access-lnmc8" (OuterVolumeSpecName: "kube-api-access-lnmc8") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "kube-api-access-lnmc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:59:07.602039 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.600984 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:59:07.602039 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.601357 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:59:07.602039 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.601440 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:59:07.602337 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.602162 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:59:07.602337 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.602295 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:59:07.603069 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.602641 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdb2801-67f8-47c8-b0c4-449daf586d34-config-out" (OuterVolumeSpecName: "config-out") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:59:07.604508 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.603332 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:59:07.605297 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.605266 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:59:07.605746 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.605729 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bdb2801-67f8-47c8-b0c4-449daf586d34-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:59:07.605995 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.605975 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-config" (OuterVolumeSpecName: "config") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:59:07.698103 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.698072 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-prometheus-k8s-tls\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.698103 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.698103 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-web-config\") pod \"8bdb2801-67f8-47c8-b0c4-449daf586d34\" (UID: \"8bdb2801-67f8-47c8-b0c4-449daf586d34\") " Jul 17 17:59:07.698304 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.698262 2572 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-kube-rbac-proxy\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.698304 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.698271 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-db\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.698304 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.698280 2572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8bdb2801-67f8-47c8-b0c4-449daf586d34-config-out\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.698304 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.698290 2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.698304 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.698298 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.698464 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.698307 2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.698464 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.698315 2572 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-config\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.698464 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.698323 2572 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-metrics-client-certs\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.698464 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.698332 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.698464 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.698340 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8bdb2801-67f8-47c8-b0c4-449daf586d34-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.698464 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.698349 2572 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-grpc-tls\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.698464 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.698357 2572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8bdb2801-67f8-47c8-b0c4-449daf586d34-tls-assets\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.698464 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.698364 2572 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.698464 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.698372 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnmc8\" (UniqueName: \"kubernetes.io/projected/8bdb2801-67f8-47c8-b0c4-449daf586d34-kube-api-access-lnmc8\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.700062 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.700039 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:59:07.708748 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.708721 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-web-config" (OuterVolumeSpecName: "web-config") pod "8bdb2801-67f8-47c8-b0c4-449daf586d34" (UID: "8bdb2801-67f8-47c8-b0c4-449daf586d34"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:59:07.798665 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.798636 2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.798665 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.798659 2572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8bdb2801-67f8-47c8-b0c4-449daf586d34-web-config\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 17:59:07.965615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.965529 2572 generic.go:356] "Generic (PLEG): container finished" podID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerID="ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983" exitCode=0 Jul 17 17:59:07.965615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.965555 2572 generic.go:356] "Generic (PLEG): container finished" podID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerID="8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b" exitCode=0 Jul 17 17:59:07.965615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.965563 2572 generic.go:356] "Generic (PLEG): container finished" podID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerID="39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc" exitCode=0 Jul 17 17:59:07.965615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.965569 2572 generic.go:356] "Generic (PLEG): container finished" podID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerID="88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda" exitCode=0 Jul 17 17:59:07.965615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.965575 2572 generic.go:356] "Generic (PLEG): container finished" podID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerID="d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf" exitCode=0 Jul 17 17:59:07.965615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.965580 2572 generic.go:356] "Generic (PLEG): container finished" podID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerID="e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177" exitCode=0 Jul 17 17:59:07.965615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.965609 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdb2801-67f8-47c8-b0c4-449daf586d34","Type":"ContainerDied","Data":"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983"} Jul 17 17:59:07.965899 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.965631 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:07.965899 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.965690 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdb2801-67f8-47c8-b0c4-449daf586d34","Type":"ContainerDied","Data":"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b"} Jul 17 17:59:07.965899 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.965719 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdb2801-67f8-47c8-b0c4-449daf586d34","Type":"ContainerDied","Data":"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc"} Jul 17 17:59:07.965899 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.965743 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdb2801-67f8-47c8-b0c4-449daf586d34","Type":"ContainerDied","Data":"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda"} Jul 17 17:59:07.965899 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.965764 2572 scope.go:117] "RemoveContainer" containerID="ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983" Jul 17 17:59:07.965899 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.965773 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdb2801-67f8-47c8-b0c4-449daf586d34","Type":"ContainerDied","Data":"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf"} Jul 17 17:59:07.965899 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.965879 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdb2801-67f8-47c8-b0c4-449daf586d34","Type":"ContainerDied","Data":"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177"} Jul 17 17:59:07.966271 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.965909 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8bdb2801-67f8-47c8-b0c4-449daf586d34","Type":"ContainerDied","Data":"5a969ac60e8c371356ea68a065f3a6a0446a0f5ba6eac6b4c1cd68f40bd948fa"} Jul 17 17:59:07.973165 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.973127 2572 scope.go:117] "RemoveContainer" containerID="8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b" Jul 17 17:59:07.979837 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.979800 2572 scope.go:117] "RemoveContainer" containerID="39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc" Jul 17 17:59:07.985981 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.985965 2572 scope.go:117] "RemoveContainer" containerID="88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda" Jul 17 17:59:07.988806 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.988782 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:59:07.992944 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.992929 2572 scope.go:117] "RemoveContainer" containerID="d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf" Jul 17 17:59:07.993949 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.993935 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:59:07.999144 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:07.999131 2572 scope.go:117] "RemoveContainer" containerID="e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177" Jul 17 17:59:08.005838 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.005823 2572 scope.go:117] "RemoveContainer" containerID="136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605" Jul 17 17:59:08.011947 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.011931 2572 scope.go:117] "RemoveContainer" containerID="ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983" Jul 17 17:59:08.012206 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:59:08.012186 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983\": container with ID starting with ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983 not found: ID does not exist" containerID="ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983" Jul 17 17:59:08.012283 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.012213 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983"} err="failed to get container status \"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983\": rpc error: code = NotFound desc = could not find container \"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983\": container with ID starting with ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983 not found: ID does not exist" Jul 17 17:59:08.012283 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.012232 2572 scope.go:117] "RemoveContainer" containerID="8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b" Jul 17 17:59:08.012428 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:59:08.012413 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b\": container with ID starting with 8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b not found: ID does not exist" containerID="8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b" Jul 17 17:59:08.012472 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.012431 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b"} err="failed to get container status \"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b\": rpc error: code = NotFound desc = could not find container \"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b\": container with ID starting with 8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b not found: ID does not exist" Jul 17 17:59:08.012472 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.012444 2572 scope.go:117] "RemoveContainer" containerID="39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc" Jul 17 17:59:08.012632 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:59:08.012617 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc\": container with ID starting with 39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc not found: ID does not exist" containerID="39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc" Jul 17 17:59:08.012674 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.012635 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc"} err="failed to get container status \"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc\": rpc error: code = NotFound desc = could not find container \"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc\": container with ID starting with 39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc not found: ID does not exist" Jul 17 17:59:08.012674 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.012649 2572 scope.go:117] "RemoveContainer" containerID="88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda" Jul 17 17:59:08.012868 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:59:08.012853 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda\": container with ID starting with 88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda not found: ID does not exist" containerID="88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda" Jul 17 17:59:08.012905 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.012873 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda"} err="failed to get container status \"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda\": rpc error: code = NotFound desc = could not find container \"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda\": container with ID starting with 88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda not found: ID does not exist" Jul 17 17:59:08.012905 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.012886 2572 scope.go:117] "RemoveContainer" containerID="d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf" Jul 17 17:59:08.013121 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:59:08.013107 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf\": container with ID starting with d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf not found: ID does not exist" containerID="d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf" Jul 17 17:59:08.013163 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.013124 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf"} err="failed to get container status \"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf\": rpc error: code = NotFound desc = could not find container \"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf\": container with ID starting with d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf not found: ID does not exist" Jul 17 17:59:08.013163 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.013145 2572 scope.go:117] "RemoveContainer" containerID="e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177" Jul 17 17:59:08.013352 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:59:08.013334 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177\": container with ID starting with e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177 not found: ID does not exist" containerID="e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177" Jul 17 17:59:08.013389 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.013359 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177"} err="failed to get container status \"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177\": rpc error: code = NotFound desc = could not find container \"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177\": container with ID starting with e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177 not found: ID does not exist" Jul 17 17:59:08.013389 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.013375 2572 scope.go:117] "RemoveContainer" containerID="136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605" Jul 17 17:59:08.013590 ip-10-0-132-62 kubenswrapper[2572]: E0717 17:59:08.013576 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605\": container with ID starting with 136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605 not found: ID does not exist" containerID="136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605" Jul 17 17:59:08.013640 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.013592 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605"} err="failed to get container status \"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605\": rpc error: code = NotFound desc = could not find container \"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605\": container with ID starting with 136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605 not found: ID does not exist" Jul 17 17:59:08.013640 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.013610 2572 scope.go:117] "RemoveContainer" containerID="ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983" Jul 17 17:59:08.013803 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.013786 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983"} err="failed to get container status \"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983\": rpc error: code = NotFound desc = could not find container \"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983\": container with ID starting with ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983 not found: ID does not exist" Jul 17 17:59:08.013838 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.013804 2572 scope.go:117] "RemoveContainer" containerID="8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b" Jul 17 17:59:08.013980 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.013965 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b"} err="failed to get container status \"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b\": rpc error: code = NotFound desc = could not find container \"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b\": container with ID starting with 8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b not found: ID does not exist" Jul 17 17:59:08.014041 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.013980 2572 scope.go:117] "RemoveContainer" containerID="39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc" Jul 17 17:59:08.014189 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.014169 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc"} err="failed to get container status \"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc\": rpc error: code = NotFound desc = could not find container \"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc\": container with ID starting with 39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc not found: ID does not exist" Jul 17 17:59:08.014267 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.014192 2572 scope.go:117] "RemoveContainer" containerID="88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda" Jul 17 17:59:08.014392 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.014375 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda"} err="failed to get container status \"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda\": rpc error: code = NotFound desc = could not find container \"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda\": container with ID starting with 88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda not found: ID does not exist" Jul 17 17:59:08.014452 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.014393 2572 scope.go:117] "RemoveContainer" containerID="d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf" Jul 17 17:59:08.014599 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.014584 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf"} err="failed to get container status \"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf\": rpc error: code = NotFound desc = could not find container \"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf\": container with ID starting with d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf not found: ID does not exist" Jul 17 17:59:08.014599 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.014598 2572 scope.go:117] "RemoveContainer" containerID="e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177" Jul 17 17:59:08.014774 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.014758 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177"} err="failed to get container status \"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177\": rpc error: code = NotFound desc = could not find container \"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177\": container with ID starting with e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177 not found: ID does not exist" Jul 17 17:59:08.014816 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.014774 2572 scope.go:117] "RemoveContainer" containerID="136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605" Jul 17 17:59:08.014989 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.014973 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605"} err="failed to get container status \"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605\": rpc error: code = NotFound desc = could not find container \"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605\": container with ID starting with 136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605 not found: ID does not exist" Jul 17 17:59:08.015059 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.014990 2572 scope.go:117] "RemoveContainer" containerID="ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983" Jul 17 17:59:08.015185 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.015169 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983"} err="failed to get container status \"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983\": rpc error: code = NotFound desc = could not find container \"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983\": container with ID starting with ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983 not found: ID does not exist" Jul 17 17:59:08.015185 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.015183 2572 scope.go:117] "RemoveContainer" containerID="8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b" Jul 17 17:59:08.015378 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.015360 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b"} err="failed to get container status \"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b\": rpc error: code = NotFound desc = could not find container \"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b\": container with ID starting with 8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b not found: ID does not exist" Jul 17 17:59:08.015420 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.015378 2572 scope.go:117] "RemoveContainer" containerID="39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc" Jul 17 17:59:08.015558 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.015542 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc"} err="failed to get container status \"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc\": rpc error: code = NotFound desc = could not find container \"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc\": container with ID starting with 39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc not found: ID does not exist" Jul 17 17:59:08.015601 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.015558 2572 scope.go:117] "RemoveContainer" containerID="88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda" Jul 17 17:59:08.015759 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.015745 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda"} err="failed to get container status \"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda\": rpc error: code = NotFound desc = could not find container \"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda\": container with ID starting with 88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda not found: ID does not exist" Jul 17 17:59:08.015805 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.015760 2572 scope.go:117] "RemoveContainer" containerID="d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf" Jul 17 17:59:08.015944 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.015931 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf"} err="failed to get container status \"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf\": rpc error: code = NotFound desc = could not find container \"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf\": container with ID starting with d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf not found: ID does not exist" Jul 17 17:59:08.015985 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.015943 2572 scope.go:117] "RemoveContainer" containerID="e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177" Jul 17 17:59:08.016143 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.016128 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177"} err="failed to get container status \"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177\": rpc error: code = NotFound desc = could not find container \"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177\": container with ID starting with e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177 not found: ID does not exist" Jul 17 17:59:08.016186 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.016143 2572 scope.go:117] "RemoveContainer" containerID="136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605" Jul 17 17:59:08.016334 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.016319 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605"} err="failed to get container status \"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605\": rpc error: code = NotFound desc = could not find container \"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605\": container with ID starting with 136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605 not found: ID does not exist" Jul 17 17:59:08.016379 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.016334 2572 scope.go:117] "RemoveContainer" containerID="ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983" Jul 17 17:59:08.016530 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.016509 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983"} err="failed to get container status \"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983\": rpc error: code = NotFound desc = could not find container \"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983\": container with ID starting with ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983 not found: ID does not exist" Jul 17 17:59:08.016602 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.016531 2572 scope.go:117] "RemoveContainer" containerID="8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b" Jul 17 17:59:08.016731 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.016718 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b"} err="failed to get container status \"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b\": rpc error: code = NotFound desc = could not find container \"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b\": container with ID starting with 8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b not found: ID does not exist" Jul 17 17:59:08.016731 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.016731 2572 scope.go:117] "RemoveContainer" containerID="39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc" Jul 17 17:59:08.016887 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.016872 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc"} err="failed to get container status \"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc\": rpc error: code = NotFound desc = could not find container \"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc\": container with ID starting with 39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc not found: ID does not exist" Jul 17 17:59:08.016928 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.016889 2572 scope.go:117] "RemoveContainer" containerID="88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda" Jul 17 17:59:08.017073 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.017058 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda"} err="failed to get container status \"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda\": rpc error: code = NotFound desc = could not find container \"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda\": container with ID starting with 88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda not found: ID does not exist" Jul 17 17:59:08.017114 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.017072 2572 scope.go:117] "RemoveContainer" containerID="d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf" Jul 17 17:59:08.017233 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.017217 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf"} err="failed to get container status \"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf\": rpc error: code = NotFound desc = could not find container \"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf\": container with ID starting with d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf not found: ID does not exist" Jul 17 17:59:08.017274 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.017232 2572 scope.go:117] "RemoveContainer" containerID="e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177" Jul 17 17:59:08.017403 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.017386 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177"} err="failed to get container status \"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177\": rpc error: code = NotFound desc = could not find container \"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177\": container with ID starting with e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177 not found: ID does not exist" Jul 17 17:59:08.017461 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.017404 2572 scope.go:117] "RemoveContainer" containerID="136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605" Jul 17 17:59:08.017647 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.017629 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605"} err="failed to get container status \"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605\": rpc error: code = NotFound desc = could not find container \"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605\": container with ID starting with 136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605 not found: ID does not exist" Jul 17 17:59:08.017684 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.017656 2572 scope.go:117] "RemoveContainer" containerID="ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983" Jul 17 17:59:08.017850 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.017835 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983"} err="failed to get container status \"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983\": rpc error: code = NotFound desc = could not find container \"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983\": container with ID starting with ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983 not found: ID does not exist" Jul 17 17:59:08.017850 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.017849 2572 scope.go:117] "RemoveContainer" containerID="8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b" Jul 17 17:59:08.018079 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.018062 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b"} err="failed to get container status \"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b\": rpc error: code = NotFound desc = could not find container \"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b\": container with ID starting with 8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b not found: ID does not exist" Jul 17 17:59:08.018151 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.018080 2572 scope.go:117] "RemoveContainer" containerID="39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc" Jul 17 17:59:08.018306 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.018286 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc"} err="failed to get container status \"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc\": rpc error: code = NotFound desc = could not find container \"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc\": container with ID starting with 39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc not found: ID does not exist" Jul 17 17:59:08.018350 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.018307 2572 scope.go:117] "RemoveContainer" containerID="88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda" Jul 17 17:59:08.018496 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.018481 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda"} err="failed to get container status \"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda\": rpc error: code = NotFound desc = could not find container \"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda\": container with ID starting with 88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda not found: ID does not exist" Jul 17 17:59:08.018538 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.018496 2572 scope.go:117] "RemoveContainer" containerID="d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf" Jul 17 17:59:08.018698 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.018681 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf"} err="failed to get container status \"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf\": rpc error: code = NotFound desc = could not find container \"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf\": container with ID starting with d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf not found: ID does not exist" Jul 17 17:59:08.018761 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.018699 2572 scope.go:117] "RemoveContainer" containerID="e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177" Jul 17 17:59:08.018916 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.018900 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177"} err="failed to get container status \"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177\": rpc error: code = NotFound desc = could not find container \"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177\": container with ID starting with e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177 not found: ID does not exist" Jul 17 17:59:08.018962 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.018916 2572 scope.go:117] "RemoveContainer" containerID="136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605" Jul 17 17:59:08.019130 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.019114 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605"} err="failed to get container status \"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605\": rpc error: code = NotFound desc = could not find container \"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605\": container with ID starting with 136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605 not found: ID does not exist" Jul 17 17:59:08.019179 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.019131 2572 scope.go:117] "RemoveContainer" containerID="ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983" Jul 17 17:59:08.019337 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.019313 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983"} err="failed to get container status \"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983\": rpc error: code = NotFound desc = could not find container \"ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983\": container with ID starting with ced3363b68a0ed6d7c2637902545c8f03e6befbbcc21ee4bbfbd0782ee311983 not found: ID does not exist" Jul 17 17:59:08.019337 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.019334 2572 scope.go:117] "RemoveContainer" containerID="8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b" Jul 17 17:59:08.019692 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.019655 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b"} err="failed to get container status \"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b\": rpc error: code = NotFound desc = could not find container \"8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b\": container with ID starting with 8fe32e8eb9c19a086f197a33fe87e0d92675f0411cc8a959f95c8bf1de8a804b not found: ID does not exist" Jul 17 17:59:08.019782 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.019692 2572 scope.go:117] "RemoveContainer" containerID="39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc" Jul 17 17:59:08.020083 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.020052 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc"} err="failed to get container status \"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc\": rpc error: code = NotFound desc = could not find container \"39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc\": container with ID starting with 39ad7a7063bc04f412a333a0952ae993ff2d09e42a91612144f4f302693209fc not found: ID does not exist" Jul 17 17:59:08.020083 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.020083 2572 scope.go:117] "RemoveContainer" containerID="88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda" Jul 17 17:59:08.020366 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.020329 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda"} err="failed to get container status \"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda\": rpc error: code = NotFound desc = could not find container \"88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda\": container with ID starting with 88a8737eaf00e064248434eae0d8a0d9519c2b00473fa08640bdca81e222beda not found: ID does not exist" Jul 17 17:59:08.020366 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.020355 2572 scope.go:117] "RemoveContainer" containerID="d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf" Jul 17 17:59:08.020647 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.020628 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf"} err="failed to get container status \"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf\": rpc error: code = NotFound desc = could not find container \"d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf\": container with ID starting with d418f98dd14b5caf7ac68a64ff1816eedc42ab8071cfcc3dfaa3f4491435d5cf not found: ID does not exist" Jul 17 17:59:08.020647 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.020648 2572 scope.go:117] "RemoveContainer" containerID="e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177" Jul 17 17:59:08.020879 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.020857 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177"} err="failed to get container status \"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177\": rpc error: code = NotFound desc = could not find container \"e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177\": container with ID starting with e3eba557419cddde29dff33c28b298ff7ea34cb7debce444c3b718eff4e76177 not found: ID does not exist" Jul 17 17:59:08.020934 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.020880 2572 scope.go:117] "RemoveContainer" containerID="136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605" Jul 17 17:59:08.021167 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021147 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605"} err="failed to get container status \"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605\": rpc error: code = NotFound desc = could not find container \"136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605\": container with ID starting with 136ed73cd966b2874f359f18afed2fb67490bec5439a555de4b61024d3018605 not found: ID does not exist" Jul 17 17:59:08.021261 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021244 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:59:08.021479 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021467 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="init-config-reloader" Jul 17 17:59:08.021509 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021482 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="init-config-reloader" Jul 17 17:59:08.021509 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021497 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="kube-rbac-proxy-web" Jul 17 17:59:08.021509 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021506 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="kube-rbac-proxy-web" Jul 17 17:59:08.021615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021517 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="kube-rbac-proxy" Jul 17 17:59:08.021615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021524 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="kube-rbac-proxy" Jul 17 17:59:08.021615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021534 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="kube-rbac-proxy" Jul 17 17:59:08.021615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021539 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="kube-rbac-proxy" Jul 17 17:59:08.021615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021545 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="prom-label-proxy" Jul 17 17:59:08.021615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021550 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="prom-label-proxy" Jul 17 17:59:08.021615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021557 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="thanos-sidecar" Jul 17 17:59:08.021615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021561 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="thanos-sidecar" Jul 17 17:59:08.021615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021570 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="kube-rbac-proxy-web" Jul 17 17:59:08.021615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021575 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="kube-rbac-proxy-web" Jul 17 17:59:08.021615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021581 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="config-reloader" Jul 17 17:59:08.021615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021586 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="config-reloader" Jul 17 17:59:08.021615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021592 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="kube-rbac-proxy-thanos" Jul 17 17:59:08.021615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021597 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="kube-rbac-proxy-thanos" Jul 17 17:59:08.021615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021609 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="config-reloader" Jul 17 17:59:08.021615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021616 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="config-reloader" Jul 17 17:59:08.021615 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021622 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="prometheus" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021628 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="prometheus" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021634 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="kube-rbac-proxy-metric" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021638 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="kube-rbac-proxy-metric" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021644 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="init-config-reloader" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021649 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="init-config-reloader" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021655 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="alertmanager" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021659 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="alertmanager" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021693 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="config-reloader" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021699 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="kube-rbac-proxy" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021705 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="kube-rbac-proxy-thanos" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021710 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="kube-rbac-proxy" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021715 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="kube-rbac-proxy-metric" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021721 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="alertmanager" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021726 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="prometheus" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021730 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="thanos-sidecar" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021734 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="prom-label-proxy" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021738 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="kube-rbac-proxy-web" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021743 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" containerName="config-reloader" Jul 17 17:59:08.022091 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.021747 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c7e5de6-b30e-4800-b952-332c75398e11" containerName="kube-rbac-proxy-web" Jul 17 17:59:08.025061 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.025048 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.027420 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.027401 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 17 17:59:08.027562 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.027545 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 17 17:59:08.027634 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.027618 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-44bm4sm79vu97\"" Jul 17 17:59:08.027691 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.027640 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 17 17:59:08.027849 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.027834 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 17 17:59:08.027927 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.027909 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 17 17:59:08.028078 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.028064 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 17 17:59:08.028153 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.028071 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 17 17:59:08.028225 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.028207 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 17 17:59:08.028323 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.028310 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 17 17:59:08.028463 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.028439 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 17 17:59:08.028660 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.028644 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 17 17:59:08.028892 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.028879 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dl25j\"" Jul 17 17:59:08.029473 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.029459 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 17 17:59:08.031357 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.031337 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 17 17:59:08.034055 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.034037 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 17 17:59:08.041920 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.041902 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:59:08.101436 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101409 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.101436 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101438 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.101643 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101455 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-config\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.101643 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101478 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.101643 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101496 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.101643 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101513 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.101643 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101561 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdnq2\" (UniqueName: \"kubernetes.io/projected/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-kube-api-access-hdnq2\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.101643 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101627 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.102046 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101668 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.102046 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101692 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.102046 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101737 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.102046 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101764 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.102046 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101791 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.102046 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101822 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.102046 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101845 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.102046 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101875 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.102046 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101924 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.102046 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101944 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.102046 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.101976 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.102046 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.102001 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.202917 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.202890 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.202917 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.202921 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnq2\" (UniqueName: \"kubernetes.io/projected/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-kube-api-access-hdnq2\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.203187 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.202954 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.203187 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.202976 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.203187 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.202993 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.203187 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.203040 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.203187 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.203064 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.203187 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.203119 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.203187 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.203152 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.203187 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.203176 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.203448 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.203204 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.203448 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.203235 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.203448 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.203259 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.203448 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.203293 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.203448 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.203316 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.203692 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.203644 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.204415 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.203881 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.204415 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.203921 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.204415 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.203948 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-config\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.204415 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.204001 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.204415 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.204050 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.204415 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.204316 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.204854 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.204503 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.204914 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.204885 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.205286 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.205259 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.205511 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.205486 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.206178 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.206154 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.207057 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.207037 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.207651 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.207627 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.207977 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.207943 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.208394 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.208338 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.208394 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.208375 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.208620 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.208411 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.208620 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.208534 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.208717 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.208622 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.208717 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.208665 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.208717 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.208703 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.209212 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.209191 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-config\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.209457 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.209439 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.212390 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.212373 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdnq2\" (UniqueName: \"kubernetes.io/projected/32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb-kube-api-access-hdnq2\") pod \"prometheus-k8s-0\" (UID: \"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.334828 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.334806 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:59:08.467831 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.467804 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:59:08.470998 ip-10-0-132-62 kubenswrapper[2572]: W0717 17:59:08.470963 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c3e3ef_c984_4fe4_9bb4_5a90fcede3fb.slice/crio-11f853e47a9552d34d21d0e14fb1b3d3694726b33d4856e1063d067ea0170606 WatchSource:0}: Error finding container 11f853e47a9552d34d21d0e14fb1b3d3694726b33d4856e1063d067ea0170606: Status 404 returned error can't find the container with id 11f853e47a9552d34d21d0e14fb1b3d3694726b33d4856e1063d067ea0170606 Jul 17 17:59:08.970868 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.970831 2572 generic.go:356] "Generic (PLEG): container finished" podID="32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb" containerID="f89e0369c3c2e8fef1b32317213dfbe6eef963de4d70d5cceac54ec079d06a44" exitCode=0 Jul 17 17:59:08.971074 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.970907 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb","Type":"ContainerDied","Data":"f89e0369c3c2e8fef1b32317213dfbe6eef963de4d70d5cceac54ec079d06a44"} Jul 17 17:59:08.971074 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:08.970978 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb","Type":"ContainerStarted","Data":"11f853e47a9552d34d21d0e14fb1b3d3694726b33d4856e1063d067ea0170606"} Jul 17 17:59:09.225979 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:09.225916 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bdb2801-67f8-47c8-b0c4-449daf586d34" path="/var/lib/kubelet/pods/8bdb2801-67f8-47c8-b0c4-449daf586d34/volumes" Jul 17 17:59:09.979175 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:09.979141 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb","Type":"ContainerStarted","Data":"24b72cf681ce091f8aa3f1b493315b07f1776780d30e917085aa50c1785c3cbc"} Jul 17 17:59:09.979577 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:09.979205 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb","Type":"ContainerStarted","Data":"a82b1cc2f14d4fe51219e035e99075c15fd4791a6edeea29a1d6863d63846341"} Jul 17 17:59:09.979577 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:09.979233 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb","Type":"ContainerStarted","Data":"a0792205b59a14f18ca0adbd15bf248ca8896a7a548e13533c00898f403a8a9e"} Jul 17 17:59:09.979577 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:09.979256 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb","Type":"ContainerStarted","Data":"c7a5b26863d9cb104cc5563a8b28d68959874037c327ebfb129300b1eeaf3809"} Jul 17 17:59:09.979577 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:09.979281 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb","Type":"ContainerStarted","Data":"fb2093faf47b96d3aa39e51395327cac4e46362bc45b421ef34932d18b0eec04"} Jul 17 17:59:09.979577 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:09.979306 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb","Type":"ContainerStarted","Data":"09989defd44a159fb3283a628443a4baef7ccbc564ca60b9f258be369ce81a5b"} Jul 17 17:59:10.015353 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:10.015303 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.996486708 podStartE2EDuration="3.015288958s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.80225ms" isStatefulPod=false podCreationTimestamp="2026-07-17 17:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:59:10.011993863 +0000 UTC m=+145.421421468" watchObservedRunningTime="2026-07-17 17:59:10.015288958 +0000 UTC m=+145.424716564" Jul 17 17:59:13.335358 ip-10-0-132-62 kubenswrapper[2572]: I0717 17:59:13.335329 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 18:00:08.335178 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:08.335143 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 18:00:08.350153 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:08.350128 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 18:00:09.195177 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:09.195147 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 18:00:52.372891 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:52.372858 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fvgr4"] Jul 17 18:00:52.374970 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:52.374955 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fvgr4" Jul 17 18:00:52.377511 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:52.377486 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 18:00:52.377511 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:52.377490 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-mts79\"" Jul 17 18:00:52.378366 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:52.378352 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 18:00:52.385282 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:52.385258 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fvgr4"] Jul 17 18:00:52.513427 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:52.513400 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4772f6a-5579-43aa-848b-e5871d4709dc-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fvgr4\" (UID: \"c4772f6a-5579-43aa-848b-e5871d4709dc\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fvgr4" Jul 17 18:00:52.513583 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:52.513456 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clk45\" (UniqueName: \"kubernetes.io/projected/c4772f6a-5579-43aa-848b-e5871d4709dc-kube-api-access-clk45\") pod \"openshift-lws-operator-fd8ccff4c-fvgr4\" (UID: \"c4772f6a-5579-43aa-848b-e5871d4709dc\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fvgr4" Jul 17 18:00:52.614047 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:52.613984 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4772f6a-5579-43aa-848b-e5871d4709dc-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fvgr4\" (UID: \"c4772f6a-5579-43aa-848b-e5871d4709dc\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fvgr4" Jul 17 18:00:52.614219 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:52.614081 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clk45\" (UniqueName: \"kubernetes.io/projected/c4772f6a-5579-43aa-848b-e5871d4709dc-kube-api-access-clk45\") pod \"openshift-lws-operator-fd8ccff4c-fvgr4\" (UID: \"c4772f6a-5579-43aa-848b-e5871d4709dc\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fvgr4" Jul 17 18:00:52.614366 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:52.614347 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4772f6a-5579-43aa-848b-e5871d4709dc-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fvgr4\" (UID: \"c4772f6a-5579-43aa-848b-e5871d4709dc\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fvgr4" Jul 17 18:00:52.623350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:52.623290 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clk45\" (UniqueName: \"kubernetes.io/projected/c4772f6a-5579-43aa-848b-e5871d4709dc-kube-api-access-clk45\") pod \"openshift-lws-operator-fd8ccff4c-fvgr4\" (UID: \"c4772f6a-5579-43aa-848b-e5871d4709dc\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fvgr4" Jul 17 18:00:52.695242 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:52.695206 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fvgr4" Jul 17 18:00:52.821425 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:52.821401 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fvgr4"] Jul 17 18:00:52.823946 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:00:52.823920 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4772f6a_5579_43aa_848b_e5871d4709dc.slice/crio-7443b452a0a2d1c59b4818b182072105b7b49969d8517609c706f2ca8f4e8f03 WatchSource:0}: Error finding container 7443b452a0a2d1c59b4818b182072105b7b49969d8517609c706f2ca8f4e8f03: Status 404 returned error can't find the container with id 7443b452a0a2d1c59b4818b182072105b7b49969d8517609c706f2ca8f4e8f03 Jul 17 18:00:53.327680 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:53.327644 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fvgr4" event={"ID":"c4772f6a-5579-43aa-848b-e5871d4709dc","Type":"ContainerStarted","Data":"7443b452a0a2d1c59b4818b182072105b7b49969d8517609c706f2ca8f4e8f03"} Jul 17 18:00:56.342225 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:56.342190 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fvgr4" event={"ID":"c4772f6a-5579-43aa-848b-e5871d4709dc","Type":"ContainerStarted","Data":"1f83579a886e4b20a01df296cdbde0f84fcffa349a731bfd25456194c8af78ed"} Jul 17 18:00:56.363603 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:00:56.363559 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fvgr4" podStartSLOduration=1.702790953 podStartE2EDuration="4.363542997s" totalImagesPullingTime="2.660752044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:00:56.362242534 +0000 UTC m=+251.771670142" watchObservedRunningTime="2026-07-17 18:00:56.363542997 +0000 UTC m=+251.772970603" Jul 17 18:01:12.028602 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.028531 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8"] Jul 17 18:01:12.031507 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.031492 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:12.034185 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.034162 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 18:01:12.034300 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.034255 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 18:01:12.034300 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.034275 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 17 18:01:12.035103 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.035080 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p5jsc\"" Jul 17 18:01:12.035232 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.035085 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 18:01:12.052162 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.052143 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8"] Jul 17 18:01:12.061115 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.061093 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/378e4b36-e952-4175-80f4-5756ab57c91a-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d8bdcf84-f82c8\" (UID: \"378e4b36-e952-4175-80f4-5756ab57c91a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:12.061228 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.061140 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lptw\" (UniqueName: \"kubernetes.io/projected/378e4b36-e952-4175-80f4-5756ab57c91a-kube-api-access-8lptw\") pod \"opendatahub-operator-controller-manager-5d8bdcf84-f82c8\" (UID: \"378e4b36-e952-4175-80f4-5756ab57c91a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:12.061228 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.061200 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/378e4b36-e952-4175-80f4-5756ab57c91a-tmp\") pod \"opendatahub-operator-controller-manager-5d8bdcf84-f82c8\" (UID: \"378e4b36-e952-4175-80f4-5756ab57c91a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:12.061313 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.061270 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/378e4b36-e952-4175-80f4-5756ab57c91a-manifests\") pod \"opendatahub-operator-controller-manager-5d8bdcf84-f82c8\" (UID: \"378e4b36-e952-4175-80f4-5756ab57c91a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:12.061358 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.061306 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/378e4b36-e952-4175-80f4-5756ab57c91a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d8bdcf84-f82c8\" (UID: \"378e4b36-e952-4175-80f4-5756ab57c91a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:12.161861 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.161837 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/378e4b36-e952-4175-80f4-5756ab57c91a-manifests\") pod \"opendatahub-operator-controller-manager-5d8bdcf84-f82c8\" (UID: \"378e4b36-e952-4175-80f4-5756ab57c91a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:12.162038 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.161872 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/378e4b36-e952-4175-80f4-5756ab57c91a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d8bdcf84-f82c8\" (UID: \"378e4b36-e952-4175-80f4-5756ab57c91a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:12.162038 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.161912 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/378e4b36-e952-4175-80f4-5756ab57c91a-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d8bdcf84-f82c8\" (UID: \"378e4b36-e952-4175-80f4-5756ab57c91a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:12.162038 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.161937 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lptw\" (UniqueName: \"kubernetes.io/projected/378e4b36-e952-4175-80f4-5756ab57c91a-kube-api-access-8lptw\") pod \"opendatahub-operator-controller-manager-5d8bdcf84-f82c8\" (UID: \"378e4b36-e952-4175-80f4-5756ab57c91a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:12.162038 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.161964 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/378e4b36-e952-4175-80f4-5756ab57c91a-tmp\") pod \"opendatahub-operator-controller-manager-5d8bdcf84-f82c8\" (UID: \"378e4b36-e952-4175-80f4-5756ab57c91a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:12.162327 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.162302 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/378e4b36-e952-4175-80f4-5756ab57c91a-manifests\") pod \"opendatahub-operator-controller-manager-5d8bdcf84-f82c8\" (UID: \"378e4b36-e952-4175-80f4-5756ab57c91a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:12.164230 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.164203 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/378e4b36-e952-4175-80f4-5756ab57c91a-tmp\") pod \"opendatahub-operator-controller-manager-5d8bdcf84-f82c8\" (UID: \"378e4b36-e952-4175-80f4-5756ab57c91a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:12.164363 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.164344 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/378e4b36-e952-4175-80f4-5756ab57c91a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d8bdcf84-f82c8\" (UID: \"378e4b36-e952-4175-80f4-5756ab57c91a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:12.164427 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.164409 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/378e4b36-e952-4175-80f4-5756ab57c91a-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d8bdcf84-f82c8\" (UID: \"378e4b36-e952-4175-80f4-5756ab57c91a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:12.174307 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.174287 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lptw\" (UniqueName: \"kubernetes.io/projected/378e4b36-e952-4175-80f4-5756ab57c91a-kube-api-access-8lptw\") pod \"opendatahub-operator-controller-manager-5d8bdcf84-f82c8\" (UID: \"378e4b36-e952-4175-80f4-5756ab57c91a\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:12.342042 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.341964 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:12.472374 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:12.472349 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8"] Jul 17 18:01:12.474755 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:01:12.474735 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378e4b36_e952_4175_80f4_5756ab57c91a.slice/crio-d7eb5029f41127e1afa77f8fdbb45cdcdc1996ff4f5ca9d7cfd0d3caecbd421a WatchSource:0}: Error finding container d7eb5029f41127e1afa77f8fdbb45cdcdc1996ff4f5ca9d7cfd0d3caecbd421a: Status 404 returned error can't find the container with id d7eb5029f41127e1afa77f8fdbb45cdcdc1996ff4f5ca9d7cfd0d3caecbd421a Jul 17 18:01:13.411655 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:13.411576 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" event={"ID":"378e4b36-e952-4175-80f4-5756ab57c91a","Type":"ContainerStarted","Data":"d7eb5029f41127e1afa77f8fdbb45cdcdc1996ff4f5ca9d7cfd0d3caecbd421a"} Jul 17 18:01:16.425025 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:16.424985 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" event={"ID":"378e4b36-e952-4175-80f4-5756ab57c91a","Type":"ContainerStarted","Data":"2598eaa5f633aa09f8a82d08fef8b609a256e21c2742b7a21cf67708556f3ea1"} Jul 17 18:01:17.430146 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:17.430108 2572 generic.go:356] "Generic (PLEG): container finished" podID="378e4b36-e952-4175-80f4-5756ab57c91a" containerID="2598eaa5f633aa09f8a82d08fef8b609a256e21c2742b7a21cf67708556f3ea1" exitCode=0 Jul 17 18:01:17.430515 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:17.430186 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" event={"ID":"378e4b36-e952-4175-80f4-5756ab57c91a","Type":"ContainerDied","Data":"2598eaa5f633aa09f8a82d08fef8b609a256e21c2742b7a21cf67708556f3ea1"} Jul 17 18:01:18.435098 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:18.434992 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" event={"ID":"378e4b36-e952-4175-80f4-5756ab57c91a","Type":"ContainerStarted","Data":"603cf906e1f1635d12bb17f18d87967a9d8d19f74712a5cdfc24ab71387a64f9"} Jul 17 18:01:18.435439 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:18.435129 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:18.457520 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:18.457468 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" podStartSLOduration=1.704893767 podStartE2EDuration="6.457451561s" totalImagesPullingTime="3.679516519s" totalInitContainerRuntime="1.073041275s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:01:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:01:18.455239018 +0000 UTC m=+273.864666624" watchObservedRunningTime="2026-07-17 18:01:18.457451561 +0000 UTC m=+273.866879167" Jul 17 18:01:29.440290 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:29.440259 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d8bdcf84-f82c8" Jul 17 18:01:32.509712 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:32.509683 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84bfff449b-t7wpv"] Jul 17 18:01:32.521228 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:32.521209 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84bfff449b-t7wpv" Jul 17 18:01:32.525224 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:32.525201 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 18:01:32.525863 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:32.525818 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 17 18:01:32.525965 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:32.525926 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 17 18:01:32.526075 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:32.525949 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 18:01:32.526234 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:32.526215 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-r5k6d\"" Jul 17 18:01:32.527593 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:32.527573 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84bfff449b-t7wpv"] Jul 17 18:01:32.629484 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:32.629453 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23012aad-30af-44b0-af8e-f341abe9d845-tls-certs\") pod \"kube-auth-proxy-84bfff449b-t7wpv\" (UID: \"23012aad-30af-44b0-af8e-f341abe9d845\") " pod="openshift-ingress/kube-auth-proxy-84bfff449b-t7wpv" Jul 17 18:01:32.629618 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:32.629490 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wd7n\" (UniqueName: \"kubernetes.io/projected/23012aad-30af-44b0-af8e-f341abe9d845-kube-api-access-6wd7n\") pod \"kube-auth-proxy-84bfff449b-t7wpv\" (UID: \"23012aad-30af-44b0-af8e-f341abe9d845\") " pod="openshift-ingress/kube-auth-proxy-84bfff449b-t7wpv" Jul 17 18:01:32.629618 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:32.629530 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23012aad-30af-44b0-af8e-f341abe9d845-tmp\") pod \"kube-auth-proxy-84bfff449b-t7wpv\" (UID: \"23012aad-30af-44b0-af8e-f341abe9d845\") " pod="openshift-ingress/kube-auth-proxy-84bfff449b-t7wpv" Jul 17 18:01:32.730777 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:32.730742 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23012aad-30af-44b0-af8e-f341abe9d845-tls-certs\") pod \"kube-auth-proxy-84bfff449b-t7wpv\" (UID: \"23012aad-30af-44b0-af8e-f341abe9d845\") " pod="openshift-ingress/kube-auth-proxy-84bfff449b-t7wpv" Jul 17 18:01:32.730931 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:32.730779 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wd7n\" (UniqueName: \"kubernetes.io/projected/23012aad-30af-44b0-af8e-f341abe9d845-kube-api-access-6wd7n\") pod \"kube-auth-proxy-84bfff449b-t7wpv\" (UID: \"23012aad-30af-44b0-af8e-f341abe9d845\") " pod="openshift-ingress/kube-auth-proxy-84bfff449b-t7wpv" Jul 17 18:01:32.730931 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:32.730810 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23012aad-30af-44b0-af8e-f341abe9d845-tmp\") pod \"kube-auth-proxy-84bfff449b-t7wpv\" (UID: \"23012aad-30af-44b0-af8e-f341abe9d845\") " pod="openshift-ingress/kube-auth-proxy-84bfff449b-t7wpv" Jul 17 18:01:32.730931 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:01:32.730892 2572 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 17 18:01:32.731121 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:01:32.730968 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23012aad-30af-44b0-af8e-f341abe9d845-tls-certs podName:23012aad-30af-44b0-af8e-f341abe9d845 nodeName:}" failed. No retries permitted until 2026-07-17 18:01:33.23095175 +0000 UTC m=+288.640379335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/23012aad-30af-44b0-af8e-f341abe9d845-tls-certs") pod "kube-auth-proxy-84bfff449b-t7wpv" (UID: "23012aad-30af-44b0-af8e-f341abe9d845") : secret "kube-auth-proxy-tls" not found Jul 17 18:01:32.732952 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:32.732934 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23012aad-30af-44b0-af8e-f341abe9d845-tmp\") pod \"kube-auth-proxy-84bfff449b-t7wpv\" (UID: \"23012aad-30af-44b0-af8e-f341abe9d845\") " pod="openshift-ingress/kube-auth-proxy-84bfff449b-t7wpv" Jul 17 18:01:32.739814 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:32.739793 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wd7n\" (UniqueName: \"kubernetes.io/projected/23012aad-30af-44b0-af8e-f341abe9d845-kube-api-access-6wd7n\") pod \"kube-auth-proxy-84bfff449b-t7wpv\" (UID: \"23012aad-30af-44b0-af8e-f341abe9d845\") " pod="openshift-ingress/kube-auth-proxy-84bfff449b-t7wpv" Jul 17 18:01:33.235365 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:33.235333 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23012aad-30af-44b0-af8e-f341abe9d845-tls-certs\") pod \"kube-auth-proxy-84bfff449b-t7wpv\" (UID: \"23012aad-30af-44b0-af8e-f341abe9d845\") " pod="openshift-ingress/kube-auth-proxy-84bfff449b-t7wpv" Jul 17 18:01:33.237611 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:33.237589 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23012aad-30af-44b0-af8e-f341abe9d845-tls-certs\") pod \"kube-auth-proxy-84bfff449b-t7wpv\" (UID: \"23012aad-30af-44b0-af8e-f341abe9d845\") " pod="openshift-ingress/kube-auth-proxy-84bfff449b-t7wpv" Jul 17 18:01:33.432099 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:33.432070 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84bfff449b-t7wpv" Jul 17 18:01:33.575690 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:33.575652 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84bfff449b-t7wpv"] Jul 17 18:01:33.582361 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:01:33.582336 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23012aad_30af_44b0_af8e_f341abe9d845.slice/crio-9dd47c98da27d6cdb20f336037f090f4eed86d9c2c056ce417b8342b0750dee1 WatchSource:0}: Error finding container 9dd47c98da27d6cdb20f336037f090f4eed86d9c2c056ce417b8342b0750dee1: Status 404 returned error can't find the container with id 9dd47c98da27d6cdb20f336037f090f4eed86d9c2c056ce417b8342b0750dee1 Jul 17 18:01:34.492775 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:34.492736 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84bfff449b-t7wpv" event={"ID":"23012aad-30af-44b0-af8e-f341abe9d845","Type":"ContainerStarted","Data":"9dd47c98da27d6cdb20f336037f090f4eed86d9c2c056ce417b8342b0750dee1"} Jul 17 18:01:36.507369 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:36.507282 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84bfff449b-t7wpv" event={"ID":"23012aad-30af-44b0-af8e-f341abe9d845","Type":"ContainerStarted","Data":"40684b64fdaa04e7a88d0efd27eb717da2dd249f77ba2dabcb32f0405f5375a2"} Jul 17 18:01:36.528545 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:36.528494 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84bfff449b-t7wpv" podStartSLOduration=1.98269729 podStartE2EDuration="4.528478572s" totalImagesPullingTime="2.545781282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:01:36.525530716 +0000 UTC m=+291.934958342" watchObservedRunningTime="2026-07-17 18:01:36.528478572 +0000 UTC m=+291.937906206" Jul 17 18:01:40.125988 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.125955 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z"] Jul 17 18:01:40.128991 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.128975 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.131644 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.131622 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 17 18:01:40.131889 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.131872 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 17 18:01:40.131973 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.131872 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 17 18:01:40.131973 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.131906 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h2ctv\"" Jul 17 18:01:40.132099 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.132059 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 17 18:01:40.143385 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.143362 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z"] Jul 17 18:01:40.192458 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.192424 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.192607 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.192474 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.192607 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.192511 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.192607 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.192540 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.192607 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.192571 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grggr\" (UniqueName: \"kubernetes.io/projected/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-kube-api-access-grggr\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.192750 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.192670 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.192750 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.192706 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.293220 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.293194 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.293354 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.293229 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.293354 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.293251 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.293354 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.293266 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.293354 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.293280 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grggr\" (UniqueName: \"kubernetes.io/projected/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-kube-api-access-grggr\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.293558 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.293472 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.293558 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.293520 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.294161 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.294131 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.295654 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.295628 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.295654 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.295636 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.295769 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.295727 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.295804 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.295776 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.301815 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.301788 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.302058 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.302042 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grggr\" (UniqueName: \"kubernetes.io/projected/aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76-kube-api-access-grggr\") pod \"istiod-openshift-gateway-94bb8fbfd-7rf9z\" (UID: \"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.437988 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.437923 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:40.575071 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:40.575039 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z"] Jul 17 18:01:40.577824 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:01:40.577799 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7d4a7f_c6d2_4b1e_808d_a1e44b37bd76.slice/crio-b230da13be77866746f19f40f75ad0433167582cfb1008c9eae4c4f995abdea5 WatchSource:0}: Error finding container b230da13be77866746f19f40f75ad0433167582cfb1008c9eae4c4f995abdea5: Status 404 returned error can't find the container with id b230da13be77866746f19f40f75ad0433167582cfb1008c9eae4c4f995abdea5 Jul 17 18:01:41.527261 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:41.527212 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" event={"ID":"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76","Type":"ContainerStarted","Data":"b230da13be77866746f19f40f75ad0433167582cfb1008c9eae4c4f995abdea5"} Jul 17 18:01:42.843630 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:42.843585 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:01:42.843918 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:42.843669 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 17 18:01:43.169432 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.169344 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8"] Jul 17 18:01:43.172698 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.172679 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" Jul 17 18:01:43.177456 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.177435 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 17 18:01:43.177804 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.177785 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-28vnl\"" Jul 17 18:01:43.178413 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.178394 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 17 18:01:43.178794 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.178754 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 17 18:01:43.197895 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.197868 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8"] Jul 17 18:01:43.319138 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.319110 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27b397ee-fdd1-43d2-8439-7396211b1acd-cert\") pod \"lws-controller-manager-f76c48b9d-9gpn8\" (UID: \"27b397ee-fdd1-43d2-8439-7396211b1acd\") " pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" Jul 17 18:01:43.319318 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.319162 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/27b397ee-fdd1-43d2-8439-7396211b1acd-metrics-cert\") pod \"lws-controller-manager-f76c48b9d-9gpn8\" (UID: \"27b397ee-fdd1-43d2-8439-7396211b1acd\") " pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" Jul 17 18:01:43.319318 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.319248 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/27b397ee-fdd1-43d2-8439-7396211b1acd-manager-config\") pod \"lws-controller-manager-f76c48b9d-9gpn8\" (UID: \"27b397ee-fdd1-43d2-8439-7396211b1acd\") " pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" Jul 17 18:01:43.319431 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.319317 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lhx7\" (UniqueName: \"kubernetes.io/projected/27b397ee-fdd1-43d2-8439-7396211b1acd-kube-api-access-9lhx7\") pod \"lws-controller-manager-f76c48b9d-9gpn8\" (UID: \"27b397ee-fdd1-43d2-8439-7396211b1acd\") " pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" Jul 17 18:01:43.420132 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.420052 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27b397ee-fdd1-43d2-8439-7396211b1acd-cert\") pod \"lws-controller-manager-f76c48b9d-9gpn8\" (UID: \"27b397ee-fdd1-43d2-8439-7396211b1acd\") " pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" Jul 17 18:01:43.420132 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.420109 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/27b397ee-fdd1-43d2-8439-7396211b1acd-metrics-cert\") pod \"lws-controller-manager-f76c48b9d-9gpn8\" (UID: \"27b397ee-fdd1-43d2-8439-7396211b1acd\") " pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" Jul 17 18:01:43.420355 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.420159 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/27b397ee-fdd1-43d2-8439-7396211b1acd-manager-config\") pod \"lws-controller-manager-f76c48b9d-9gpn8\" (UID: \"27b397ee-fdd1-43d2-8439-7396211b1acd\") " pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" Jul 17 18:01:43.420355 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.420188 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lhx7\" (UniqueName: \"kubernetes.io/projected/27b397ee-fdd1-43d2-8439-7396211b1acd-kube-api-access-9lhx7\") pod \"lws-controller-manager-f76c48b9d-9gpn8\" (UID: \"27b397ee-fdd1-43d2-8439-7396211b1acd\") " pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" Jul 17 18:01:43.420847 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.420827 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/27b397ee-fdd1-43d2-8439-7396211b1acd-manager-config\") pod \"lws-controller-manager-f76c48b9d-9gpn8\" (UID: \"27b397ee-fdd1-43d2-8439-7396211b1acd\") " pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" Jul 17 18:01:43.422523 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.422495 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27b397ee-fdd1-43d2-8439-7396211b1acd-cert\") pod \"lws-controller-manager-f76c48b9d-9gpn8\" (UID: \"27b397ee-fdd1-43d2-8439-7396211b1acd\") " pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" Jul 17 18:01:43.422896 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.422876 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/27b397ee-fdd1-43d2-8439-7396211b1acd-metrics-cert\") pod \"lws-controller-manager-f76c48b9d-9gpn8\" (UID: \"27b397ee-fdd1-43d2-8439-7396211b1acd\") " pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" Jul 17 18:01:43.441165 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.441140 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lhx7\" (UniqueName: \"kubernetes.io/projected/27b397ee-fdd1-43d2-8439-7396211b1acd-kube-api-access-9lhx7\") pod \"lws-controller-manager-f76c48b9d-9gpn8\" (UID: \"27b397ee-fdd1-43d2-8439-7396211b1acd\") " pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" Jul 17 18:01:43.481755 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.481728 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" Jul 17 18:01:43.539039 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.538953 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" event={"ID":"aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76","Type":"ContainerStarted","Data":"e0e62c048168bfef2804cf395721ea94ad9d5c56df25457cdbcaaf368d00d3c0"} Jul 17 18:01:43.539518 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.539487 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:43.540963 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.540939 2572 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-7rf9z container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 18:01:43.541056 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.540991 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" podUID="aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 18:01:43.565868 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.565787 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" podStartSLOduration=1.3022039300000001 podStartE2EDuration="3.565766206s" totalImagesPullingTime="2.263562276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:01:43.564411064 +0000 UTC m=+298.973838673" watchObservedRunningTime="2026-07-17 18:01:43.565766206 +0000 UTC m=+298.975193812" Jul 17 18:01:43.633233 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:43.633200 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8"] Jul 17 18:01:43.636186 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:01:43.636152 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b397ee_fdd1_43d2_8439_7396211b1acd.slice/crio-b25693c50ba370d53d0687064e0ceca5bbe2ddf14b525e594a3d3677d3148ecb WatchSource:0}: Error finding container b25693c50ba370d53d0687064e0ceca5bbe2ddf14b525e594a3d3677d3148ecb: Status 404 returned error can't find the container with id b25693c50ba370d53d0687064e0ceca5bbe2ddf14b525e594a3d3677d3148ecb Jul 17 18:01:44.544081 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:44.544050 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" event={"ID":"27b397ee-fdd1-43d2-8439-7396211b1acd","Type":"ContainerStarted","Data":"b25693c50ba370d53d0687064e0ceca5bbe2ddf14b525e594a3d3677d3148ecb"} Jul 17 18:01:44.544833 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:44.544814 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7rf9z" Jul 17 18:01:45.098960 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:45.098932 2572 kubelet.go:1648] "Image garbage collection succeeded" Jul 17 18:01:46.554182 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:46.554143 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" event={"ID":"27b397ee-fdd1-43d2-8439-7396211b1acd","Type":"ContainerStarted","Data":"68ad621589c829f6f2ad3cf44ba22238b00447fd04520c7a6c7ae1b73257db33"} Jul 17 18:01:46.554589 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:46.554246 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" Jul 17 18:01:46.581507 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:46.581456 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" podStartSLOduration=1.151485748 podStartE2EDuration="3.58144031s" totalImagesPullingTime="2.429954562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:01:46.579700546 +0000 UTC m=+301.989128166" watchObservedRunningTime="2026-07-17 18:01:46.58144031 +0000 UTC m=+301.990867916" Jul 17 18:01:57.560799 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:01:57.560771 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-f76c48b9d-9gpn8" Jul 17 18:06:40.586514 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.586477 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8"] Jul 17 18:06:40.589689 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.589668 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.592181 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.592162 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 17 18:06:40.593215 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.593199 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 17 18:06:40.593383 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.593362 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gsgfc\"" Jul 17 18:06:40.593486 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.593427 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 17 18:06:40.600115 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.600095 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8"] Jul 17 18:06:40.685897 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.685857 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vqhs8\" (UID: \"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.686183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.686164 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vqhs8\" (UID: \"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.686306 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.686293 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vqhs8\" (UID: \"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.686420 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.686407 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq6q4\" (UniqueName: \"kubernetes.io/projected/f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7-kube-api-access-pq6q4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vqhs8\" (UID: \"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.686651 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.686629 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vqhs8\" (UID: \"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.686935 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.686910 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vqhs8\" (UID: \"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.788087 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.788053 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vqhs8\" (UID: \"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.788087 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.788101 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vqhs8\" (UID: \"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.788312 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.788148 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq6q4\" (UniqueName: \"kubernetes.io/projected/f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7-kube-api-access-pq6q4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vqhs8\" (UID: \"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.788312 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.788189 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vqhs8\" (UID: \"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.788312 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.788251 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vqhs8\" (UID: \"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.788441 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.788322 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vqhs8\" (UID: \"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.788678 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.788649 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vqhs8\" (UID: \"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.788678 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.788668 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vqhs8\" (UID: \"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.788790 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.788732 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vqhs8\" (UID: \"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.790358 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.790337 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vqhs8\" (UID: \"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.790733 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.790710 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vqhs8\" (UID: \"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.795712 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.795694 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq6q4\" (UniqueName: \"kubernetes.io/projected/f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7-kube-api-access-pq6q4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vqhs8\" (UID: \"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:40.900738 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:40.900641 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:41.024346 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:41.024241 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8"] Jul 17 18:06:41.026505 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:06:41.026480 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33ef96e_58b0_4e4a_b80c_48b1d55cb8b7.slice/crio-813c62193fa854a5f7b9ef9ec26249d76eea5f5fa7a06314f93acd1812694ace WatchSource:0}: Error finding container 813c62193fa854a5f7b9ef9ec26249d76eea5f5fa7a06314f93acd1812694ace: Status 404 returned error can't find the container with id 813c62193fa854a5f7b9ef9ec26249d76eea5f5fa7a06314f93acd1812694ace Jul 17 18:06:41.027799 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:41.027784 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:06:41.699339 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:41.699303 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" event={"ID":"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7","Type":"ContainerStarted","Data":"813c62193fa854a5f7b9ef9ec26249d76eea5f5fa7a06314f93acd1812694ace"} Jul 17 18:06:42.704889 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:42.704852 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" event={"ID":"f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7","Type":"ContainerStarted","Data":"8d11a9abe59ff7723fd65620895a74cd5415232d26dd665eb158aa5bc398ec7d"} Jul 17 18:06:42.724456 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:42.724408 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" podStartSLOduration=1.653911071 podStartE2EDuration="2.724390395s" totalImagesPullingTime="1.070479324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:06:42.722879779 +0000 UTC m=+598.132307386" watchObservedRunningTime="2026-07-17 18:06:42.724390395 +0000 UTC m=+598.133818001" Jul 17 18:06:50.900880 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:50.900831 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:50.900880 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:50.900900 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:50.913358 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:50.913327 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:06:51.755453 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:06:51.755427 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vqhs8" Jul 17 18:27:56.736024 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:27:56.735974 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl"] Jul 17 18:27:56.739363 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:27:56.739347 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl" Jul 17 18:27:56.741989 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:27:56.741955 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:27:56.742127 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:27:56.742028 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:27:56.747573 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:27:56.747551 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl"] Jul 17 18:27:56.818393 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:27:56.818364 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b54803f-9335-4465-a370-d09a80b06016-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl\" (UID: \"3b54803f-9335-4465-a370-d09a80b06016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl" Jul 17 18:27:56.919543 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:27:56.919499 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b54803f-9335-4465-a370-d09a80b06016-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl\" (UID: \"3b54803f-9335-4465-a370-d09a80b06016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl" Jul 17 18:27:56.920215 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:27:56.920191 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b54803f-9335-4465-a370-d09a80b06016-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl\" (UID: \"3b54803f-9335-4465-a370-d09a80b06016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl" Jul 17 18:27:57.050153 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:27:57.050124 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl" Jul 17 18:27:57.198680 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:27:57.198644 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl"] Jul 17 18:27:57.201536 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:27:57.201510 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b54803f_9335_4465_a370_d09a80b06016.slice/crio-1072b553ccb136cd8450c9f0ff2a83474c7b4f85dc1aed11ea9e4dada5d48b0a WatchSource:0}: Error finding container 1072b553ccb136cd8450c9f0ff2a83474c7b4f85dc1aed11ea9e4dada5d48b0a: Status 404 returned error can't find the container with id 1072b553ccb136cd8450c9f0ff2a83474c7b4f85dc1aed11ea9e4dada5d48b0a Jul 17 18:27:57.203292 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:27:57.203274 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:27:57.788619 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:27:57.788585 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl" event={"ID":"3b54803f-9335-4465-a370-d09a80b06016","Type":"ContainerStarted","Data":"1072b553ccb136cd8450c9f0ff2a83474c7b4f85dc1aed11ea9e4dada5d48b0a"} Jul 17 18:28:00.804909 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:00.804818 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl" event={"ID":"3b54803f-9335-4465-a370-d09a80b06016","Type":"ContainerStarted","Data":"6ce2796bdf9cc2fd40fc2d094bb769e9a89d9acd7040c1bb3bf34e2d44abe409"} Jul 17 18:28:00.822495 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:00.822445 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl" podStartSLOduration=1.7405490719999999 podStartE2EDuration="4.822429357s" totalImagesPullingTime="3.081880285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:27:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:28:00.820505554 +0000 UTC m=+1876.229933160" watchObservedRunningTime="2026-07-17 18:28:00.822429357 +0000 UTC m=+1876.231857014" Jul 17 18:28:01.811226 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:01.811192 2572 generic.go:356] "Generic (PLEG): container finished" podID="3b54803f-9335-4465-a370-d09a80b06016" containerID="6ce2796bdf9cc2fd40fc2d094bb769e9a89d9acd7040c1bb3bf34e2d44abe409" exitCode=6 Jul 17 18:28:01.811671 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:01.811281 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl" event={"ID":"3b54803f-9335-4465-a370-d09a80b06016","Type":"ContainerDied","Data":"6ce2796bdf9cc2fd40fc2d094bb769e9a89d9acd7040c1bb3bf34e2d44abe409"} Jul 17 18:28:01.811671 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:01.811459 2572 scope.go:117] "RemoveContainer" containerID="6ce2796bdf9cc2fd40fc2d094bb769e9a89d9acd7040c1bb3bf34e2d44abe409" Jul 17 18:28:02.817496 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:02.817402 2572 generic.go:356] "Generic (PLEG): container finished" podID="3b54803f-9335-4465-a370-d09a80b06016" containerID="97175065d6e51d1bce0cd94693ccbaba16c7a980cb798ca7db62885582ed31c6" exitCode=6 Jul 17 18:28:02.817496 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:02.817481 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl" event={"ID":"3b54803f-9335-4465-a370-d09a80b06016","Type":"ContainerDied","Data":"97175065d6e51d1bce0cd94693ccbaba16c7a980cb798ca7db62885582ed31c6"} Jul 17 18:28:02.817921 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:02.817539 2572 scope.go:117] "RemoveContainer" containerID="6ce2796bdf9cc2fd40fc2d094bb769e9a89d9acd7040c1bb3bf34e2d44abe409" Jul 17 18:28:02.817921 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:02.817746 2572 scope.go:117] "RemoveContainer" containerID="97175065d6e51d1bce0cd94693ccbaba16c7a980cb798ca7db62885582ed31c6" Jul 17 18:28:02.817994 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:28:02.817946 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl_odh-ai-gateway-infra(3b54803f-9335-4465-a370-d09a80b06016)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl" podUID="3b54803f-9335-4465-a370-d09a80b06016" Jul 17 18:28:03.823530 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:03.823502 2572 scope.go:117] "RemoveContainer" containerID="97175065d6e51d1bce0cd94693ccbaba16c7a980cb798ca7db62885582ed31c6" Jul 17 18:28:03.823904 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:28:03.823692 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl_odh-ai-gateway-infra(3b54803f-9335-4465-a370-d09a80b06016)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl" podUID="3b54803f-9335-4465-a370-d09a80b06016" Jul 17 18:28:19.219396 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:19.219326 2572 scope.go:117] "RemoveContainer" containerID="97175065d6e51d1bce0cd94693ccbaba16c7a980cb798ca7db62885582ed31c6" Jul 17 18:28:19.898028 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:19.897990 2572 generic.go:356] "Generic (PLEG): container finished" podID="3b54803f-9335-4465-a370-d09a80b06016" containerID="9d3f341dba5c97de4e7e849be7f1261688bd8a2e68928aec5c63e02fbae5579a" exitCode=6 Jul 17 18:28:19.898146 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:19.898069 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl" event={"ID":"3b54803f-9335-4465-a370-d09a80b06016","Type":"ContainerDied","Data":"9d3f341dba5c97de4e7e849be7f1261688bd8a2e68928aec5c63e02fbae5579a"} Jul 17 18:28:19.898146 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:19.898130 2572 scope.go:117] "RemoveContainer" containerID="97175065d6e51d1bce0cd94693ccbaba16c7a980cb798ca7db62885582ed31c6" Jul 17 18:28:19.898432 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:19.898408 2572 scope.go:117] "RemoveContainer" containerID="9d3f341dba5c97de4e7e849be7f1261688bd8a2e68928aec5c63e02fbae5579a" Jul 17 18:28:19.898656 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:28:19.898637 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl_odh-ai-gateway-infra(3b54803f-9335-4465-a370-d09a80b06016)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl" podUID="3b54803f-9335-4465-a370-d09a80b06016" Jul 17 18:28:20.922858 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:20.922824 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl"] Jul 17 18:28:21.045642 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:21.045622 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl" Jul 17 18:28:21.122680 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:21.122650 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b54803f-9335-4465-a370-d09a80b06016-maas-api-service-ca\") pod \"3b54803f-9335-4465-a370-d09a80b06016\" (UID: \"3b54803f-9335-4465-a370-d09a80b06016\") " Jul 17 18:28:21.122980 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:21.122958 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b54803f-9335-4465-a370-d09a80b06016-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b54803f-9335-4465-a370-d09a80b06016" (UID: "3b54803f-9335-4465-a370-d09a80b06016"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:28:21.224365 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:21.224271 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b54803f-9335-4465-a370-d09a80b06016-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:28:21.910129 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:21.910093 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl" event={"ID":"3b54803f-9335-4465-a370-d09a80b06016","Type":"ContainerDied","Data":"1072b553ccb136cd8450c9f0ff2a83474c7b4f85dc1aed11ea9e4dada5d48b0a"} Jul 17 18:28:21.910129 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:21.910127 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl" Jul 17 18:28:21.910360 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:21.910160 2572 scope.go:117] "RemoveContainer" containerID="9d3f341dba5c97de4e7e849be7f1261688bd8a2e68928aec5c63e02fbae5579a" Jul 17 18:28:21.925195 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:21.925169 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl"] Jul 17 18:28:21.928857 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:21.928837 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-jn5kl"] Jul 17 18:28:23.224057 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:23.224004 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b54803f-9335-4465-a370-d09a80b06016" path="/var/lib/kubelet/pods/3b54803f-9335-4465-a370-d09a80b06016/volumes" Jul 17 18:28:42.801422 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:42.801388 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m"] Jul 17 18:28:42.801898 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:42.801696 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b54803f-9335-4465-a370-d09a80b06016" containerName="revoke-keys" Jul 17 18:28:42.801898 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:42.801706 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b54803f-9335-4465-a370-d09a80b06016" containerName="revoke-keys" Jul 17 18:28:42.801898 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:42.801718 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b54803f-9335-4465-a370-d09a80b06016" containerName="revoke-keys" Jul 17 18:28:42.801898 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:42.801725 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b54803f-9335-4465-a370-d09a80b06016" containerName="revoke-keys" Jul 17 18:28:42.801898 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:42.801736 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b54803f-9335-4465-a370-d09a80b06016" containerName="revoke-keys" Jul 17 18:28:42.801898 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:42.801741 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b54803f-9335-4465-a370-d09a80b06016" containerName="revoke-keys" Jul 17 18:28:42.801898 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:42.801789 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b54803f-9335-4465-a370-d09a80b06016" containerName="revoke-keys" Jul 17 18:28:42.801898 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:42.801798 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b54803f-9335-4465-a370-d09a80b06016" containerName="revoke-keys" Jul 17 18:28:42.809636 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:42.809611 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m" Jul 17 18:28:42.811625 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:42.811597 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m"] Jul 17 18:28:42.812397 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:42.812372 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:28:42.812522 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:42.812371 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:28:42.901528 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:42.901497 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50bb163-9d1e-4fd4-87f9-5cb6716034f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m\" (UID: \"d50bb163-9d1e-4fd4-87f9-5cb6716034f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m" Jul 17 18:28:43.002750 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:43.002724 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50bb163-9d1e-4fd4-87f9-5cb6716034f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m\" (UID: \"d50bb163-9d1e-4fd4-87f9-5cb6716034f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m" Jul 17 18:28:43.003343 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:43.003327 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50bb163-9d1e-4fd4-87f9-5cb6716034f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m\" (UID: \"d50bb163-9d1e-4fd4-87f9-5cb6716034f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m" Jul 17 18:28:43.120612 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:43.120521 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m" Jul 17 18:28:43.242762 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:43.242617 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m"] Jul 17 18:28:43.244810 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:28:43.244782 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50bb163_9d1e_4fd4_87f9_5cb6716034f3.slice/crio-48386ccf8c83b0fec86a83e7e89d150d4ac53dacc28cf71a8aa4d67019753171 WatchSource:0}: Error finding container 48386ccf8c83b0fec86a83e7e89d150d4ac53dacc28cf71a8aa4d67019753171: Status 404 returned error can't find the container with id 48386ccf8c83b0fec86a83e7e89d150d4ac53dacc28cf71a8aa4d67019753171 Jul 17 18:28:43.997876 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:43.997845 2572 generic.go:356] "Generic (PLEG): container finished" podID="d50bb163-9d1e-4fd4-87f9-5cb6716034f3" containerID="25c547af31643310dc0a86d211b26b1c63a66fca1dbcb1660ce7511fdc814184" exitCode=7 Jul 17 18:28:43.998240 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:43.997931 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m" event={"ID":"d50bb163-9d1e-4fd4-87f9-5cb6716034f3","Type":"ContainerDied","Data":"25c547af31643310dc0a86d211b26b1c63a66fca1dbcb1660ce7511fdc814184"} Jul 17 18:28:43.998240 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:43.997987 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m" event={"ID":"d50bb163-9d1e-4fd4-87f9-5cb6716034f3","Type":"ContainerStarted","Data":"48386ccf8c83b0fec86a83e7e89d150d4ac53dacc28cf71a8aa4d67019753171"} Jul 17 18:28:43.998240 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:43.998107 2572 scope.go:117] "RemoveContainer" containerID="25c547af31643310dc0a86d211b26b1c63a66fca1dbcb1660ce7511fdc814184" Jul 17 18:28:45.005737 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:45.005643 2572 generic.go:356] "Generic (PLEG): container finished" podID="d50bb163-9d1e-4fd4-87f9-5cb6716034f3" containerID="7c230862c857c6f78be38b89c9c5380720a8fab56a2134ce8adef0cf3f51f0da" exitCode=7 Jul 17 18:28:45.006189 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:45.005728 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m" event={"ID":"d50bb163-9d1e-4fd4-87f9-5cb6716034f3","Type":"ContainerDied","Data":"7c230862c857c6f78be38b89c9c5380720a8fab56a2134ce8adef0cf3f51f0da"} Jul 17 18:28:45.006189 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:45.005887 2572 scope.go:117] "RemoveContainer" containerID="25c547af31643310dc0a86d211b26b1c63a66fca1dbcb1660ce7511fdc814184" Jul 17 18:28:45.006189 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:45.006118 2572 scope.go:117] "RemoveContainer" containerID="7c230862c857c6f78be38b89c9c5380720a8fab56a2134ce8adef0cf3f51f0da" Jul 17 18:28:45.006380 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:28:45.006356 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m_odh-ai-gateway-infra(d50bb163-9d1e-4fd4-87f9-5cb6716034f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m" podUID="d50bb163-9d1e-4fd4-87f9-5cb6716034f3" Jul 17 18:28:46.011713 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:46.011688 2572 scope.go:117] "RemoveContainer" containerID="7c230862c857c6f78be38b89c9c5380720a8fab56a2134ce8adef0cf3f51f0da" Jul 17 18:28:46.012109 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:28:46.011894 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m_odh-ai-gateway-infra(d50bb163-9d1e-4fd4-87f9-5cb6716034f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m" podUID="d50bb163-9d1e-4fd4-87f9-5cb6716034f3" Jul 17 18:28:56.779355 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:56.779325 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5"] Jul 17 18:28:56.779722 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:56.779660 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b54803f-9335-4465-a370-d09a80b06016" containerName="revoke-keys" Jul 17 18:28:56.786500 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:56.786475 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5" Jul 17 18:28:56.790429 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:56.790400 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5"] Jul 17 18:28:56.926974 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:56.926944 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e8bb31-e53f-4e59-b289-d0f478160874-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-njps5\" (UID: \"d1e8bb31-e53f-4e59-b289-d0f478160874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5" Jul 17 18:28:57.028082 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:57.028044 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e8bb31-e53f-4e59-b289-d0f478160874-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-njps5\" (UID: \"d1e8bb31-e53f-4e59-b289-d0f478160874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5" Jul 17 18:28:57.028650 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:57.028631 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e8bb31-e53f-4e59-b289-d0f478160874-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-njps5\" (UID: \"d1e8bb31-e53f-4e59-b289-d0f478160874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5" Jul 17 18:28:57.098855 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:57.098781 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5" Jul 17 18:28:57.226903 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:57.226829 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5"] Jul 17 18:28:57.229572 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:28:57.229545 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e8bb31_e53f_4e59_b289_d0f478160874.slice/crio-5063585d6f5e5b195d6d0601e89c4d65411b074cb76998b3b7a52f5fef87df1f WatchSource:0}: Error finding container 5063585d6f5e5b195d6d0601e89c4d65411b074cb76998b3b7a52f5fef87df1f: Status 404 returned error can't find the container with id 5063585d6f5e5b195d6d0601e89c4d65411b074cb76998b3b7a52f5fef87df1f Jul 17 18:28:58.059959 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:58.059923 2572 generic.go:356] "Generic (PLEG): container finished" podID="d1e8bb31-e53f-4e59-b289-d0f478160874" containerID="02f6d9ed61458c97e80ecdbe643ff55281dd88fd37bb2100d6808f72083157ae" exitCode=6 Jul 17 18:28:58.060361 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:58.060024 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5" event={"ID":"d1e8bb31-e53f-4e59-b289-d0f478160874","Type":"ContainerDied","Data":"02f6d9ed61458c97e80ecdbe643ff55281dd88fd37bb2100d6808f72083157ae"} Jul 17 18:28:58.060361 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:58.060079 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5" event={"ID":"d1e8bb31-e53f-4e59-b289-d0f478160874","Type":"ContainerStarted","Data":"5063585d6f5e5b195d6d0601e89c4d65411b074cb76998b3b7a52f5fef87df1f"} Jul 17 18:28:58.060361 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:58.060248 2572 scope.go:117] "RemoveContainer" containerID="02f6d9ed61458c97e80ecdbe643ff55281dd88fd37bb2100d6808f72083157ae" Jul 17 18:28:59.065854 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:59.065825 2572 generic.go:356] "Generic (PLEG): container finished" podID="d1e8bb31-e53f-4e59-b289-d0f478160874" containerID="3e19ba2781afc9a6d9dce14299c34b8f38f657d1062805bbad7eec064ec7a81a" exitCode=6 Jul 17 18:28:59.066366 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:59.065902 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5" event={"ID":"d1e8bb31-e53f-4e59-b289-d0f478160874","Type":"ContainerDied","Data":"3e19ba2781afc9a6d9dce14299c34b8f38f657d1062805bbad7eec064ec7a81a"} Jul 17 18:28:59.066366 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:59.065949 2572 scope.go:117] "RemoveContainer" containerID="02f6d9ed61458c97e80ecdbe643ff55281dd88fd37bb2100d6808f72083157ae" Jul 17 18:28:59.066366 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:59.066193 2572 scope.go:117] "RemoveContainer" containerID="3e19ba2781afc9a6d9dce14299c34b8f38f657d1062805bbad7eec064ec7a81a" Jul 17 18:28:59.066519 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:28:59.066391 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-njps5_odh-ai-gateway-infra(d1e8bb31-e53f-4e59-b289-d0f478160874)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5" podUID="d1e8bb31-e53f-4e59-b289-d0f478160874" Jul 17 18:28:59.220315 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:28:59.220285 2572 scope.go:117] "RemoveContainer" containerID="7c230862c857c6f78be38b89c9c5380720a8fab56a2134ce8adef0cf3f51f0da" Jul 17 18:29:00.071949 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:00.071917 2572 generic.go:356] "Generic (PLEG): container finished" podID="d50bb163-9d1e-4fd4-87f9-5cb6716034f3" containerID="e65827ba49504e386c104dd7fd095099dc2bb816c81ac37c46c42242bbf858a8" exitCode=7 Jul 17 18:29:00.072404 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:00.071984 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m" event={"ID":"d50bb163-9d1e-4fd4-87f9-5cb6716034f3","Type":"ContainerDied","Data":"e65827ba49504e386c104dd7fd095099dc2bb816c81ac37c46c42242bbf858a8"} Jul 17 18:29:00.072404 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:00.072062 2572 scope.go:117] "RemoveContainer" containerID="7c230862c857c6f78be38b89c9c5380720a8fab56a2134ce8adef0cf3f51f0da" Jul 17 18:29:00.072404 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:00.072363 2572 scope.go:117] "RemoveContainer" containerID="e65827ba49504e386c104dd7fd095099dc2bb816c81ac37c46c42242bbf858a8" Jul 17 18:29:00.072610 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:29:00.072591 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m_odh-ai-gateway-infra(d50bb163-9d1e-4fd4-87f9-5cb6716034f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m" podUID="d50bb163-9d1e-4fd4-87f9-5cb6716034f3" Jul 17 18:29:00.073708 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:00.073691 2572 scope.go:117] "RemoveContainer" containerID="3e19ba2781afc9a6d9dce14299c34b8f38f657d1062805bbad7eec064ec7a81a" Jul 17 18:29:00.073870 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:29:00.073852 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-njps5_odh-ai-gateway-infra(d1e8bb31-e53f-4e59-b289-d0f478160874)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5" podUID="d1e8bb31-e53f-4e59-b289-d0f478160874" Jul 17 18:29:01.096950 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:01.096917 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m"] Jul 17 18:29:01.225488 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:01.225464 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m" Jul 17 18:29:01.264026 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:01.263985 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50bb163-9d1e-4fd4-87f9-5cb6716034f3-maas-api-service-ca\") pod \"d50bb163-9d1e-4fd4-87f9-5cb6716034f3\" (UID: \"d50bb163-9d1e-4fd4-87f9-5cb6716034f3\") " Jul 17 18:29:01.264342 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:01.264316 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50bb163-9d1e-4fd4-87f9-5cb6716034f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d50bb163-9d1e-4fd4-87f9-5cb6716034f3" (UID: "d50bb163-9d1e-4fd4-87f9-5cb6716034f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:29:01.364791 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:01.364717 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d50bb163-9d1e-4fd4-87f9-5cb6716034f3-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:29:02.083984 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:02.083952 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m" event={"ID":"d50bb163-9d1e-4fd4-87f9-5cb6716034f3","Type":"ContainerDied","Data":"48386ccf8c83b0fec86a83e7e89d150d4ac53dacc28cf71a8aa4d67019753171"} Jul 17 18:29:02.084199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:02.083965 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m" Jul 17 18:29:02.084199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:02.084049 2572 scope.go:117] "RemoveContainer" containerID="e65827ba49504e386c104dd7fd095099dc2bb816c81ac37c46c42242bbf858a8" Jul 17 18:29:02.104469 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:02.104447 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m"] Jul 17 18:29:02.114721 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:02.114700 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-qhz6m"] Jul 17 18:29:03.224445 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:03.224415 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d50bb163-9d1e-4fd4-87f9-5cb6716034f3" path="/var/lib/kubelet/pods/d50bb163-9d1e-4fd4-87f9-5cb6716034f3/volumes" Jul 17 18:29:04.627288 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:04.627259 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9"] Jul 17 18:29:04.627629 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:04.627537 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50bb163-9d1e-4fd4-87f9-5cb6716034f3" containerName="revoke-keys" Jul 17 18:29:04.627629 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:04.627547 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50bb163-9d1e-4fd4-87f9-5cb6716034f3" containerName="revoke-keys" Jul 17 18:29:04.627629 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:04.627562 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50bb163-9d1e-4fd4-87f9-5cb6716034f3" containerName="revoke-keys" Jul 17 18:29:04.627629 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:04.627569 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50bb163-9d1e-4fd4-87f9-5cb6716034f3" containerName="revoke-keys" Jul 17 18:29:04.627629 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:04.627582 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50bb163-9d1e-4fd4-87f9-5cb6716034f3" containerName="revoke-keys" Jul 17 18:29:04.627629 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:04.627588 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50bb163-9d1e-4fd4-87f9-5cb6716034f3" containerName="revoke-keys" Jul 17 18:29:04.627824 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:04.627637 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d50bb163-9d1e-4fd4-87f9-5cb6716034f3" containerName="revoke-keys" Jul 17 18:29:04.627824 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:04.627645 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d50bb163-9d1e-4fd4-87f9-5cb6716034f3" containerName="revoke-keys" Jul 17 18:29:04.627824 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:04.627652 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d50bb163-9d1e-4fd4-87f9-5cb6716034f3" containerName="revoke-keys" Jul 17 18:29:04.631841 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:04.631825 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9" Jul 17 18:29:04.638648 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:04.638623 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9"] Jul 17 18:29:04.687049 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:04.686999 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f44b8c0-4403-456f-9d9d-f02bead32860-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9\" (UID: \"3f44b8c0-4403-456f-9d9d-f02bead32860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9" Jul 17 18:29:04.787924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:04.787897 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f44b8c0-4403-456f-9d9d-f02bead32860-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9\" (UID: \"3f44b8c0-4403-456f-9d9d-f02bead32860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9" Jul 17 18:29:04.788502 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:04.788485 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f44b8c0-4403-456f-9d9d-f02bead32860-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9\" (UID: \"3f44b8c0-4403-456f-9d9d-f02bead32860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9" Jul 17 18:29:04.942817 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:04.942735 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9" Jul 17 18:29:05.066976 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:05.066951 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9"] Jul 17 18:29:05.069639 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:29:05.069614 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f44b8c0_4403_456f_9d9d_f02bead32860.slice/crio-08502b5edb333b602b22a316271bc0cb670aad867f8c283a58196a339cecc5fc WatchSource:0}: Error finding container 08502b5edb333b602b22a316271bc0cb670aad867f8c283a58196a339cecc5fc: Status 404 returned error can't find the container with id 08502b5edb333b602b22a316271bc0cb670aad867f8c283a58196a339cecc5fc Jul 17 18:29:05.097492 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:05.097465 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9" event={"ID":"3f44b8c0-4403-456f-9d9d-f02bead32860","Type":"ContainerStarted","Data":"08502b5edb333b602b22a316271bc0cb670aad867f8c283a58196a339cecc5fc"} Jul 17 18:29:06.103690 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:06.103655 2572 generic.go:356] "Generic (PLEG): container finished" podID="3f44b8c0-4403-456f-9d9d-f02bead32860" containerID="d1642a248546cb7540e2d28f9f15b54ea421cfe066bddc80791e6349dc5c652b" exitCode=6 Jul 17 18:29:06.104078 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:06.103720 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9" event={"ID":"3f44b8c0-4403-456f-9d9d-f02bead32860","Type":"ContainerDied","Data":"d1642a248546cb7540e2d28f9f15b54ea421cfe066bddc80791e6349dc5c652b"} Jul 17 18:29:06.104078 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:06.103901 2572 scope.go:117] "RemoveContainer" containerID="d1642a248546cb7540e2d28f9f15b54ea421cfe066bddc80791e6349dc5c652b" Jul 17 18:29:07.115568 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:07.115484 2572 generic.go:356] "Generic (PLEG): container finished" podID="3f44b8c0-4403-456f-9d9d-f02bead32860" containerID="2a307bdbc7eaab79a732b51ffa4ee6c95339762c3705fbcc8ef05e8e47b0732f" exitCode=6 Jul 17 18:29:07.115937 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:07.115565 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9" event={"ID":"3f44b8c0-4403-456f-9d9d-f02bead32860","Type":"ContainerDied","Data":"2a307bdbc7eaab79a732b51ffa4ee6c95339762c3705fbcc8ef05e8e47b0732f"} Jul 17 18:29:07.115937 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:07.115648 2572 scope.go:117] "RemoveContainer" containerID="d1642a248546cb7540e2d28f9f15b54ea421cfe066bddc80791e6349dc5c652b" Jul 17 18:29:07.115937 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:07.115865 2572 scope.go:117] "RemoveContainer" containerID="2a307bdbc7eaab79a732b51ffa4ee6c95339762c3705fbcc8ef05e8e47b0732f" Jul 17 18:29:07.116112 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:29:07.116089 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9_odh-ai-gateway-infra(3f44b8c0-4403-456f-9d9d-f02bead32860)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9" podUID="3f44b8c0-4403-456f-9d9d-f02bead32860" Jul 17 18:29:08.122165 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:08.122140 2572 scope.go:117] "RemoveContainer" containerID="2a307bdbc7eaab79a732b51ffa4ee6c95339762c3705fbcc8ef05e8e47b0732f" Jul 17 18:29:08.122527 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:29:08.122323 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9_odh-ai-gateway-infra(3f44b8c0-4403-456f-9d9d-f02bead32860)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9" podUID="3f44b8c0-4403-456f-9d9d-f02bead32860" Jul 17 18:29:14.220200 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:14.220167 2572 scope.go:117] "RemoveContainer" containerID="3e19ba2781afc9a6d9dce14299c34b8f38f657d1062805bbad7eec064ec7a81a" Jul 17 18:29:15.154178 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:15.154092 2572 generic.go:356] "Generic (PLEG): container finished" podID="d1e8bb31-e53f-4e59-b289-d0f478160874" containerID="49599d01e631add38c41278d0cc37c8c4a229b31a666878ca541f743a886657f" exitCode=6 Jul 17 18:29:15.154178 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:15.154168 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5" event={"ID":"d1e8bb31-e53f-4e59-b289-d0f478160874","Type":"ContainerDied","Data":"49599d01e631add38c41278d0cc37c8c4a229b31a666878ca541f743a886657f"} Jul 17 18:29:15.154370 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:15.154254 2572 scope.go:117] "RemoveContainer" containerID="3e19ba2781afc9a6d9dce14299c34b8f38f657d1062805bbad7eec064ec7a81a" Jul 17 18:29:15.154605 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:15.154579 2572 scope.go:117] "RemoveContainer" containerID="49599d01e631add38c41278d0cc37c8c4a229b31a666878ca541f743a886657f" Jul 17 18:29:15.154830 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:29:15.154812 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-njps5_odh-ai-gateway-infra(d1e8bb31-e53f-4e59-b289-d0f478160874)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5" podUID="d1e8bb31-e53f-4e59-b289-d0f478160874" Jul 17 18:29:16.178271 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:16.178234 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5"] Jul 17 18:29:16.310718 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:16.310697 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5" Jul 17 18:29:16.381348 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:16.381317 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e8bb31-e53f-4e59-b289-d0f478160874-maas-api-service-ca\") pod \"d1e8bb31-e53f-4e59-b289-d0f478160874\" (UID: \"d1e8bb31-e53f-4e59-b289-d0f478160874\") " Jul 17 18:29:16.381653 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:16.381632 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e8bb31-e53f-4e59-b289-d0f478160874-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d1e8bb31-e53f-4e59-b289-d0f478160874" (UID: "d1e8bb31-e53f-4e59-b289-d0f478160874"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:29:16.482075 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:16.481971 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e8bb31-e53f-4e59-b289-d0f478160874-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:29:17.165539 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:17.165508 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5" Jul 17 18:29:17.165719 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:17.165508 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5" event={"ID":"d1e8bb31-e53f-4e59-b289-d0f478160874","Type":"ContainerDied","Data":"5063585d6f5e5b195d6d0601e89c4d65411b074cb76998b3b7a52f5fef87df1f"} Jul 17 18:29:17.165719 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:17.165636 2572 scope.go:117] "RemoveContainer" containerID="49599d01e631add38c41278d0cc37c8c4a229b31a666878ca541f743a886657f" Jul 17 18:29:17.185942 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:17.185917 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5"] Jul 17 18:29:17.193672 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:17.193642 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-njps5"] Jul 17 18:29:17.223873 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:17.223843 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1e8bb31-e53f-4e59-b289-d0f478160874" path="/var/lib/kubelet/pods/d1e8bb31-e53f-4e59-b289-d0f478160874/volumes" Jul 17 18:29:23.220154 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:23.220122 2572 scope.go:117] "RemoveContainer" containerID="2a307bdbc7eaab79a732b51ffa4ee6c95339762c3705fbcc8ef05e8e47b0732f" Jul 17 18:29:24.198397 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:24.198316 2572 generic.go:356] "Generic (PLEG): container finished" podID="3f44b8c0-4403-456f-9d9d-f02bead32860" containerID="a5baa160a5c852eda6e5eab35901e2bc48631247237e7bebb70a5610e7b58033" exitCode=6 Jul 17 18:29:24.198397 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:24.198361 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9" event={"ID":"3f44b8c0-4403-456f-9d9d-f02bead32860","Type":"ContainerDied","Data":"a5baa160a5c852eda6e5eab35901e2bc48631247237e7bebb70a5610e7b58033"} Jul 17 18:29:24.198581 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:24.198409 2572 scope.go:117] "RemoveContainer" containerID="2a307bdbc7eaab79a732b51ffa4ee6c95339762c3705fbcc8ef05e8e47b0732f" Jul 17 18:29:24.198702 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:24.198685 2572 scope.go:117] "RemoveContainer" containerID="a5baa160a5c852eda6e5eab35901e2bc48631247237e7bebb70a5610e7b58033" Jul 17 18:29:24.198918 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:29:24.198892 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9_odh-ai-gateway-infra(3f44b8c0-4403-456f-9d9d-f02bead32860)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9" podUID="3f44b8c0-4403-456f-9d9d-f02bead32860" Jul 17 18:29:25.224414 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:25.224384 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9"] Jul 17 18:29:25.351112 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:25.351090 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9" Jul 17 18:29:25.455231 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:25.455206 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f44b8c0-4403-456f-9d9d-f02bead32860-maas-api-service-ca\") pod \"3f44b8c0-4403-456f-9d9d-f02bead32860\" (UID: \"3f44b8c0-4403-456f-9d9d-f02bead32860\") " Jul 17 18:29:25.455564 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:25.455541 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f44b8c0-4403-456f-9d9d-f02bead32860-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f44b8c0-4403-456f-9d9d-f02bead32860" (UID: "3f44b8c0-4403-456f-9d9d-f02bead32860"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:29:25.556234 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:25.556193 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f44b8c0-4403-456f-9d9d-f02bead32860-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:29:26.208816 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:26.208787 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9" Jul 17 18:29:26.209047 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:26.208787 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9" event={"ID":"3f44b8c0-4403-456f-9d9d-f02bead32860","Type":"ContainerDied","Data":"08502b5edb333b602b22a316271bc0cb670aad867f8c283a58196a339cecc5fc"} Jul 17 18:29:26.209047 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:26.208945 2572 scope.go:117] "RemoveContainer" containerID="a5baa160a5c852eda6e5eab35901e2bc48631247237e7bebb70a5610e7b58033" Jul 17 18:29:26.228645 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:26.228624 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9"] Jul 17 18:29:26.235111 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:26.235089 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-s9vd9"] Jul 17 18:29:27.224155 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:27.224127 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f44b8c0-4403-456f-9d9d-f02bead32860" path="/var/lib/kubelet/pods/3f44b8c0-4403-456f-9d9d-f02bead32860/volumes" Jul 17 18:29:32.847802 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.847771 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw"] Jul 17 18:29:32.848195 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.848092 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f44b8c0-4403-456f-9d9d-f02bead32860" containerName="revoke-keys" Jul 17 18:29:32.848195 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.848105 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f44b8c0-4403-456f-9d9d-f02bead32860" containerName="revoke-keys" Jul 17 18:29:32.848195 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.848121 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e8bb31-e53f-4e59-b289-d0f478160874" containerName="revoke-keys" Jul 17 18:29:32.848195 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.848126 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e8bb31-e53f-4e59-b289-d0f478160874" containerName="revoke-keys" Jul 17 18:29:32.848195 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.848138 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f44b8c0-4403-456f-9d9d-f02bead32860" containerName="revoke-keys" Jul 17 18:29:32.848195 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.848144 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f44b8c0-4403-456f-9d9d-f02bead32860" containerName="revoke-keys" Jul 17 18:29:32.848195 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.848153 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f44b8c0-4403-456f-9d9d-f02bead32860" containerName="revoke-keys" Jul 17 18:29:32.848195 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.848158 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f44b8c0-4403-456f-9d9d-f02bead32860" containerName="revoke-keys" Jul 17 18:29:32.848195 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.848168 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e8bb31-e53f-4e59-b289-d0f478160874" containerName="revoke-keys" Jul 17 18:29:32.848195 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.848174 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e8bb31-e53f-4e59-b289-d0f478160874" containerName="revoke-keys" Jul 17 18:29:32.848484 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.848218 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e8bb31-e53f-4e59-b289-d0f478160874" containerName="revoke-keys" Jul 17 18:29:32.848484 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.848227 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f44b8c0-4403-456f-9d9d-f02bead32860" containerName="revoke-keys" Jul 17 18:29:32.848484 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.848232 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f44b8c0-4403-456f-9d9d-f02bead32860" containerName="revoke-keys" Jul 17 18:29:32.848484 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.848238 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e8bb31-e53f-4e59-b289-d0f478160874" containerName="revoke-keys" Jul 17 18:29:32.848484 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.848244 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f44b8c0-4403-456f-9d9d-f02bead32860" containerName="revoke-keys" Jul 17 18:29:32.852736 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.852715 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw" Jul 17 18:29:32.855386 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.855371 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:29:32.855473 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.855374 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:29:32.858996 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.858969 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw"] Jul 17 18:29:32.915624 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:32.915599 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbda39b0-50e1-4fc9-92a8-a3516de6d410-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw\" (UID: \"bbda39b0-50e1-4fc9-92a8-a3516de6d410\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw" Jul 17 18:29:33.016653 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:33.016627 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbda39b0-50e1-4fc9-92a8-a3516de6d410-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw\" (UID: \"bbda39b0-50e1-4fc9-92a8-a3516de6d410\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw" Jul 17 18:29:33.017243 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:33.017225 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbda39b0-50e1-4fc9-92a8-a3516de6d410-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw\" (UID: \"bbda39b0-50e1-4fc9-92a8-a3516de6d410\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw" Jul 17 18:29:33.164407 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:33.164325 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw" Jul 17 18:29:33.290116 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:33.290082 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw"] Jul 17 18:29:33.292298 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:29:33.292270 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbda39b0_50e1_4fc9_92a8_a3516de6d410.slice/crio-fbc1fe4598ecfb62c794b976613fe8fa19591226086adf4c12cdaebe8b06ebfe WatchSource:0}: Error finding container fbc1fe4598ecfb62c794b976613fe8fa19591226086adf4c12cdaebe8b06ebfe: Status 404 returned error can't find the container with id fbc1fe4598ecfb62c794b976613fe8fa19591226086adf4c12cdaebe8b06ebfe Jul 17 18:29:34.243170 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:34.243095 2572 generic.go:356] "Generic (PLEG): container finished" podID="bbda39b0-50e1-4fc9-92a8-a3516de6d410" containerID="d415537c3b0f437c7dc83246442be6697020bf4a5327b8fd439c3982afd66c2b" exitCode=7 Jul 17 18:29:34.243539 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:34.243173 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw" event={"ID":"bbda39b0-50e1-4fc9-92a8-a3516de6d410","Type":"ContainerDied","Data":"d415537c3b0f437c7dc83246442be6697020bf4a5327b8fd439c3982afd66c2b"} Jul 17 18:29:34.243539 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:34.243227 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw" event={"ID":"bbda39b0-50e1-4fc9-92a8-a3516de6d410","Type":"ContainerStarted","Data":"fbc1fe4598ecfb62c794b976613fe8fa19591226086adf4c12cdaebe8b06ebfe"} Jul 17 18:29:34.243539 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:34.243345 2572 scope.go:117] "RemoveContainer" containerID="d415537c3b0f437c7dc83246442be6697020bf4a5327b8fd439c3982afd66c2b" Jul 17 18:29:35.248322 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:35.248233 2572 generic.go:356] "Generic (PLEG): container finished" podID="bbda39b0-50e1-4fc9-92a8-a3516de6d410" containerID="8c3ee2b446c0f1c3ee64810d96945cada77dc97df02d85b2e7518119a98e9b58" exitCode=7 Jul 17 18:29:35.248710 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:35.248315 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw" event={"ID":"bbda39b0-50e1-4fc9-92a8-a3516de6d410","Type":"ContainerDied","Data":"8c3ee2b446c0f1c3ee64810d96945cada77dc97df02d85b2e7518119a98e9b58"} Jul 17 18:29:35.248710 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:35.248373 2572 scope.go:117] "RemoveContainer" containerID="d415537c3b0f437c7dc83246442be6697020bf4a5327b8fd439c3982afd66c2b" Jul 17 18:29:35.248710 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:35.248478 2572 scope.go:117] "RemoveContainer" containerID="8c3ee2b446c0f1c3ee64810d96945cada77dc97df02d85b2e7518119a98e9b58" Jul 17 18:29:35.248710 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:29:35.248685 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw_odh-ai-gateway-infra(bbda39b0-50e1-4fc9-92a8-a3516de6d410)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw" podUID="bbda39b0-50e1-4fc9-92a8-a3516de6d410" Jul 17 18:29:36.253993 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:36.253914 2572 scope.go:117] "RemoveContainer" containerID="8c3ee2b446c0f1c3ee64810d96945cada77dc97df02d85b2e7518119a98e9b58" Jul 17 18:29:36.254471 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:29:36.254201 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw_odh-ai-gateway-infra(bbda39b0-50e1-4fc9-92a8-a3516de6d410)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw" podUID="bbda39b0-50e1-4fc9-92a8-a3516de6d410" Jul 17 18:29:48.219509 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:48.219476 2572 scope.go:117] "RemoveContainer" containerID="8c3ee2b446c0f1c3ee64810d96945cada77dc97df02d85b2e7518119a98e9b58" Jul 17 18:29:49.304588 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:49.304559 2572 generic.go:356] "Generic (PLEG): container finished" podID="bbda39b0-50e1-4fc9-92a8-a3516de6d410" containerID="25a9ea07336f561fd8b8df07fe20b425c8a3c3bc899cbd4f51cf8d350a17ca04" exitCode=7 Jul 17 18:29:49.305000 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:49.304628 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw" event={"ID":"bbda39b0-50e1-4fc9-92a8-a3516de6d410","Type":"ContainerDied","Data":"25a9ea07336f561fd8b8df07fe20b425c8a3c3bc899cbd4f51cf8d350a17ca04"} Jul 17 18:29:49.305000 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:49.304692 2572 scope.go:117] "RemoveContainer" containerID="8c3ee2b446c0f1c3ee64810d96945cada77dc97df02d85b2e7518119a98e9b58" Jul 17 18:29:49.305150 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:49.305080 2572 scope.go:117] "RemoveContainer" containerID="25a9ea07336f561fd8b8df07fe20b425c8a3c3bc899cbd4f51cf8d350a17ca04" Jul 17 18:29:49.305303 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:29:49.305279 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw_odh-ai-gateway-infra(bbda39b0-50e1-4fc9-92a8-a3516de6d410)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw" podUID="bbda39b0-50e1-4fc9-92a8-a3516de6d410" Jul 17 18:29:50.332094 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:50.332062 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw"] Jul 17 18:29:50.462781 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:50.462756 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw" Jul 17 18:29:50.577385 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:50.577359 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbda39b0-50e1-4fc9-92a8-a3516de6d410-maas-api-service-ca\") pod \"bbda39b0-50e1-4fc9-92a8-a3516de6d410\" (UID: \"bbda39b0-50e1-4fc9-92a8-a3516de6d410\") " Jul 17 18:29:50.577712 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:50.577690 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbda39b0-50e1-4fc9-92a8-a3516de6d410-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bbda39b0-50e1-4fc9-92a8-a3516de6d410" (UID: "bbda39b0-50e1-4fc9-92a8-a3516de6d410"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:29:50.678223 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:50.678150 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbda39b0-50e1-4fc9-92a8-a3516de6d410-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:29:51.315244 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:51.315204 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw" event={"ID":"bbda39b0-50e1-4fc9-92a8-a3516de6d410","Type":"ContainerDied","Data":"fbc1fe4598ecfb62c794b976613fe8fa19591226086adf4c12cdaebe8b06ebfe"} Jul 17 18:29:51.315408 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:51.315263 2572 scope.go:117] "RemoveContainer" containerID="25a9ea07336f561fd8b8df07fe20b425c8a3c3bc899cbd4f51cf8d350a17ca04" Jul 17 18:29:51.315408 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:51.315219 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw" Jul 17 18:29:51.332840 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:51.332814 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw"] Jul 17 18:29:51.336650 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:51.336627 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-zmqxw"] Jul 17 18:29:53.224289 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:29:53.224256 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbda39b0-50e1-4fc9-92a8-a3516de6d410" path="/var/lib/kubelet/pods/bbda39b0-50e1-4fc9-92a8-a3516de6d410/volumes" Jul 17 18:30:00.131951 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.131912 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738550-9vbgr"] Jul 17 18:30:00.132349 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.132246 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbda39b0-50e1-4fc9-92a8-a3516de6d410" containerName="revoke-keys" Jul 17 18:30:00.132349 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.132257 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbda39b0-50e1-4fc9-92a8-a3516de6d410" containerName="revoke-keys" Jul 17 18:30:00.132349 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.132267 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbda39b0-50e1-4fc9-92a8-a3516de6d410" containerName="revoke-keys" Jul 17 18:30:00.132349 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.132273 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbda39b0-50e1-4fc9-92a8-a3516de6d410" containerName="revoke-keys" Jul 17 18:30:00.132349 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.132284 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e8bb31-e53f-4e59-b289-d0f478160874" containerName="revoke-keys" Jul 17 18:30:00.132349 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.132289 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e8bb31-e53f-4e59-b289-d0f478160874" containerName="revoke-keys" Jul 17 18:30:00.132349 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.132342 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbda39b0-50e1-4fc9-92a8-a3516de6d410" containerName="revoke-keys" Jul 17 18:30:00.132349 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.132350 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbda39b0-50e1-4fc9-92a8-a3516de6d410" containerName="revoke-keys" Jul 17 18:30:00.132579 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.132358 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e8bb31-e53f-4e59-b289-d0f478160874" containerName="revoke-keys" Jul 17 18:30:00.136757 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.136737 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg"] Jul 17 18:30:00.136885 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.136864 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738550-9vbgr" Jul 17 18:30:00.137161 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.137139 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbda39b0-50e1-4fc9-92a8-a3516de6d410" containerName="revoke-keys" Jul 17 18:30:00.137161 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.137157 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbda39b0-50e1-4fc9-92a8-a3516de6d410" containerName="revoke-keys" Jul 17 18:30:00.137310 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.137227 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbda39b0-50e1-4fc9-92a8-a3516de6d410" containerName="revoke-keys" Jul 17 18:30:00.139389 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.139368 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:30:00.139809 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.139793 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg" Jul 17 18:30:00.142530 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.142508 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738550-9vbgr"] Jul 17 18:30:00.145748 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.145546 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg"] Jul 17 18:30:00.150288 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.150270 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738550-9vbgr" Jul 17 18:30:00.157786 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.157768 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg" Jul 17 18:30:00.284743 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.284713 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738550-9vbgr"] Jul 17 18:30:00.288045 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:30:00.287996 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915b4b0b_aa37_4cf6_855c_f52bf142a336.slice/crio-d0d6b96bde821f4191e76dc99ace4e164fbfbec2f39b8df2aec905b7d021fb9a WatchSource:0}: Error finding container d0d6b96bde821f4191e76dc99ace4e164fbfbec2f39b8df2aec905b7d021fb9a: Status 404 returned error can't find the container with id d0d6b96bde821f4191e76dc99ace4e164fbfbec2f39b8df2aec905b7d021fb9a Jul 17 18:30:00.310711 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.310686 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg"] Jul 17 18:30:00.327064 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:30:00.327041 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43b22bb_c888_443f_915b_fafbb3fe96f6.slice/crio-551eb93dc17850c0326e69d87fa11054fb492b4c432d4fe6c95e4b9d14451159 WatchSource:0}: Error finding container 551eb93dc17850c0326e69d87fa11054fb492b4c432d4fe6c95e4b9d14451159: Status 404 returned error can't find the container with id 551eb93dc17850c0326e69d87fa11054fb492b4c432d4fe6c95e4b9d14451159 Jul 17 18:30:00.353958 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.353928 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg" event={"ID":"b43b22bb-c888-443f-915b-fafbb3fe96f6","Type":"ContainerStarted","Data":"551eb93dc17850c0326e69d87fa11054fb492b4c432d4fe6c95e4b9d14451159"} Jul 17 18:30:00.354959 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:00.354940 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738550-9vbgr" event={"ID":"915b4b0b-aa37-4cf6-855c-f52bf142a336","Type":"ContainerStarted","Data":"d0d6b96bde821f4191e76dc99ace4e164fbfbec2f39b8df2aec905b7d021fb9a"} Jul 17 18:30:01.360970 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:01.360885 2572 generic.go:356] "Generic (PLEG): container finished" podID="b43b22bb-c888-443f-915b-fafbb3fe96f6" containerID="f136d519794d75b85c4ebde0f3a63ea10a15e95d4f97640ec1a915a774b7a044" exitCode=7 Jul 17 18:30:01.361407 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:01.360967 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg" event={"ID":"b43b22bb-c888-443f-915b-fafbb3fe96f6","Type":"ContainerDied","Data":"f136d519794d75b85c4ebde0f3a63ea10a15e95d4f97640ec1a915a774b7a044"} Jul 17 18:30:01.361407 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:01.361125 2572 scope.go:117] "RemoveContainer" containerID="f136d519794d75b85c4ebde0f3a63ea10a15e95d4f97640ec1a915a774b7a044" Jul 17 18:30:01.362530 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:01.362510 2572 generic.go:356] "Generic (PLEG): container finished" podID="915b4b0b-aa37-4cf6-855c-f52bf142a336" containerID="ee8f3d343642c3d3af834e8f83adf48b60ea90fc5921ba3f1add95e91e310a4f" exitCode=0 Jul 17 18:30:01.362606 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:01.362550 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738550-9vbgr" event={"ID":"915b4b0b-aa37-4cf6-855c-f52bf142a336","Type":"ContainerDied","Data":"ee8f3d343642c3d3af834e8f83adf48b60ea90fc5921ba3f1add95e91e310a4f"} Jul 17 18:30:02.370164 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:02.370074 2572 generic.go:356] "Generic (PLEG): container finished" podID="b43b22bb-c888-443f-915b-fafbb3fe96f6" containerID="c5e7806369f1527d311c3c94cb11e6ea78022e11b1f515a6a1ec78fad6da7f8c" exitCode=7 Jul 17 18:30:02.370164 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:02.370119 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg" event={"ID":"b43b22bb-c888-443f-915b-fafbb3fe96f6","Type":"ContainerDied","Data":"c5e7806369f1527d311c3c94cb11e6ea78022e11b1f515a6a1ec78fad6da7f8c"} Jul 17 18:30:02.370650 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:02.370191 2572 scope.go:117] "RemoveContainer" containerID="f136d519794d75b85c4ebde0f3a63ea10a15e95d4f97640ec1a915a774b7a044" Jul 17 18:30:02.370650 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:02.370345 2572 scope.go:117] "RemoveContainer" containerID="c5e7806369f1527d311c3c94cb11e6ea78022e11b1f515a6a1ec78fad6da7f8c" Jul 17 18:30:02.370650 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:30:02.370601 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg_odh-ai-gateway-infra(b43b22bb-c888-443f-915b-fafbb3fe96f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg" podUID="b43b22bb-c888-443f-915b-fafbb3fe96f6" Jul 17 18:30:02.496231 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:02.496209 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738550-9vbgr" Jul 17 18:30:03.375541 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:03.375511 2572 scope.go:117] "RemoveContainer" containerID="c5e7806369f1527d311c3c94cb11e6ea78022e11b1f515a6a1ec78fad6da7f8c" Jul 17 18:30:03.375928 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:30:03.375739 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg_odh-ai-gateway-infra(b43b22bb-c888-443f-915b-fafbb3fe96f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg" podUID="b43b22bb-c888-443f-915b-fafbb3fe96f6" Jul 17 18:30:03.376552 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:03.376535 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738550-9vbgr" Jul 17 18:30:03.376552 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:03.376540 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738550-9vbgr" event={"ID":"915b4b0b-aa37-4cf6-855c-f52bf142a336","Type":"ContainerDied","Data":"d0d6b96bde821f4191e76dc99ace4e164fbfbec2f39b8df2aec905b7d021fb9a"} Jul 17 18:30:03.376663 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:03.376599 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0d6b96bde821f4191e76dc99ace4e164fbfbec2f39b8df2aec905b7d021fb9a" Jul 17 18:30:04.679686 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:04.679653 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw"] Jul 17 18:30:04.680094 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:04.679976 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="915b4b0b-aa37-4cf6-855c-f52bf142a336" containerName="cleanup" Jul 17 18:30:04.680094 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:04.679988 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="915b4b0b-aa37-4cf6-855c-f52bf142a336" containerName="cleanup" Jul 17 18:30:04.680094 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:04.680054 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="915b4b0b-aa37-4cf6-855c-f52bf142a336" containerName="cleanup" Jul 17 18:30:04.682955 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:04.682939 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw" Jul 17 18:30:04.685794 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:04.685779 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:30:04.688647 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:04.688624 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw"] Jul 17 18:30:04.789619 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:04.789579 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b574a01d-cf97-4f79-887e-67665e66c7d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw\" (UID: \"b574a01d-cf97-4f79-887e-67665e66c7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw" Jul 17 18:30:04.890976 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:04.890935 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b574a01d-cf97-4f79-887e-67665e66c7d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw\" (UID: \"b574a01d-cf97-4f79-887e-67665e66c7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw" Jul 17 18:30:04.891617 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:04.891589 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b574a01d-cf97-4f79-887e-67665e66c7d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw\" (UID: \"b574a01d-cf97-4f79-887e-67665e66c7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw" Jul 17 18:30:04.995212 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:04.995138 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw" Jul 17 18:30:05.117896 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:05.117872 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw"] Jul 17 18:30:05.120633 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:30:05.120606 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb574a01d_cf97_4f79_887e_67665e66c7d4.slice/crio-0ab792ec4ff8b77b23a9bd4ed0e0861f67c95b74b2556a678ffcd0d9532867f7 WatchSource:0}: Error finding container 0ab792ec4ff8b77b23a9bd4ed0e0861f67c95b74b2556a678ffcd0d9532867f7: Status 404 returned error can't find the container with id 0ab792ec4ff8b77b23a9bd4ed0e0861f67c95b74b2556a678ffcd0d9532867f7 Jul 17 18:30:05.385399 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:05.385363 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw" event={"ID":"b574a01d-cf97-4f79-887e-67665e66c7d4","Type":"ContainerStarted","Data":"0ab792ec4ff8b77b23a9bd4ed0e0861f67c95b74b2556a678ffcd0d9532867f7"} Jul 17 18:30:06.391244 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:06.391208 2572 generic.go:356] "Generic (PLEG): container finished" podID="b574a01d-cf97-4f79-887e-67665e66c7d4" containerID="4b9336dde2aaeb09a2e44632c8259df11bdb97dca784d371115192a937628a32" exitCode=6 Jul 17 18:30:06.391619 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:06.391266 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw" event={"ID":"b574a01d-cf97-4f79-887e-67665e66c7d4","Type":"ContainerDied","Data":"4b9336dde2aaeb09a2e44632c8259df11bdb97dca784d371115192a937628a32"} Jul 17 18:30:06.391619 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:06.391465 2572 scope.go:117] "RemoveContainer" containerID="4b9336dde2aaeb09a2e44632c8259df11bdb97dca784d371115192a937628a32" Jul 17 18:30:07.396443 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:07.396358 2572 generic.go:356] "Generic (PLEG): container finished" podID="b574a01d-cf97-4f79-887e-67665e66c7d4" containerID="ef9ecc1d71a88ab998414727c3188c280815c0f22eaeaea346196a5e5674c73c" exitCode=6 Jul 17 18:30:07.396873 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:07.396444 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw" event={"ID":"b574a01d-cf97-4f79-887e-67665e66c7d4","Type":"ContainerDied","Data":"ef9ecc1d71a88ab998414727c3188c280815c0f22eaeaea346196a5e5674c73c"} Jul 17 18:30:07.396873 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:07.396499 2572 scope.go:117] "RemoveContainer" containerID="4b9336dde2aaeb09a2e44632c8259df11bdb97dca784d371115192a937628a32" Jul 17 18:30:07.396873 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:07.396630 2572 scope.go:117] "RemoveContainer" containerID="ef9ecc1d71a88ab998414727c3188c280815c0f22eaeaea346196a5e5674c73c" Jul 17 18:30:07.396873 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:30:07.396807 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw_odh-ai-gateway-infra(b574a01d-cf97-4f79-887e-67665e66c7d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw" podUID="b574a01d-cf97-4f79-887e-67665e66c7d4" Jul 17 18:30:08.402310 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:08.402279 2572 scope.go:117] "RemoveContainer" containerID="ef9ecc1d71a88ab998414727c3188c280815c0f22eaeaea346196a5e5674c73c" Jul 17 18:30:08.402749 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:30:08.402520 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw_odh-ai-gateway-infra(b574a01d-cf97-4f79-887e-67665e66c7d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw" podUID="b574a01d-cf97-4f79-887e-67665e66c7d4" Jul 17 18:30:18.220201 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:18.220169 2572 scope.go:117] "RemoveContainer" containerID="c5e7806369f1527d311c3c94cb11e6ea78022e11b1f515a6a1ec78fad6da7f8c" Jul 17 18:30:19.448106 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:19.448072 2572 generic.go:356] "Generic (PLEG): container finished" podID="b43b22bb-c888-443f-915b-fafbb3fe96f6" containerID="941759ad4496bdba558a20f58e4525b5400721aaed8f7106ac8396b542e0acba" exitCode=7 Jul 17 18:30:19.448592 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:19.448141 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg" event={"ID":"b43b22bb-c888-443f-915b-fafbb3fe96f6","Type":"ContainerDied","Data":"941759ad4496bdba558a20f58e4525b5400721aaed8f7106ac8396b542e0acba"} Jul 17 18:30:19.448592 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:19.448199 2572 scope.go:117] "RemoveContainer" containerID="c5e7806369f1527d311c3c94cb11e6ea78022e11b1f515a6a1ec78fad6da7f8c" Jul 17 18:30:19.448592 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:19.448441 2572 scope.go:117] "RemoveContainer" containerID="941759ad4496bdba558a20f58e4525b5400721aaed8f7106ac8396b542e0acba" Jul 17 18:30:19.448731 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:30:19.448656 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg_odh-ai-gateway-infra(b43b22bb-c888-443f-915b-fafbb3fe96f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg" podUID="b43b22bb-c888-443f-915b-fafbb3fe96f6" Jul 17 18:30:20.476318 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:20.476239 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg"] Jul 17 18:30:20.606105 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:20.606083 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg" Jul 17 18:30:21.219700 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:21.219668 2572 scope.go:117] "RemoveContainer" containerID="ef9ecc1d71a88ab998414727c3188c280815c0f22eaeaea346196a5e5674c73c" Jul 17 18:30:21.458551 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:21.458524 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg" Jul 17 18:30:21.458551 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:21.458536 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg" event={"ID":"b43b22bb-c888-443f-915b-fafbb3fe96f6","Type":"ContainerDied","Data":"551eb93dc17850c0326e69d87fa11054fb492b4c432d4fe6c95e4b9d14451159"} Jul 17 18:30:21.458765 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:21.458596 2572 scope.go:117] "RemoveContainer" containerID="941759ad4496bdba558a20f58e4525b5400721aaed8f7106ac8396b542e0acba" Jul 17 18:30:21.479050 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:21.478986 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg"] Jul 17 18:30:21.483481 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:21.483460 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-a5b144da-29738550-mnhhg"] Jul 17 18:30:22.470327 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:22.470293 2572 generic.go:356] "Generic (PLEG): container finished" podID="b574a01d-cf97-4f79-887e-67665e66c7d4" containerID="4369df6b031a046d2e0252c98339a4f5c8d06b24e4b2b363a59a1cb148018609" exitCode=6 Jul 17 18:30:22.470495 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:22.470345 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw" event={"ID":"b574a01d-cf97-4f79-887e-67665e66c7d4","Type":"ContainerDied","Data":"4369df6b031a046d2e0252c98339a4f5c8d06b24e4b2b363a59a1cb148018609"} Jul 17 18:30:22.470495 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:22.470414 2572 scope.go:117] "RemoveContainer" containerID="ef9ecc1d71a88ab998414727c3188c280815c0f22eaeaea346196a5e5674c73c" Jul 17 18:30:22.470762 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:22.470737 2572 scope.go:117] "RemoveContainer" containerID="4369df6b031a046d2e0252c98339a4f5c8d06b24e4b2b363a59a1cb148018609" Jul 17 18:30:22.470947 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:30:22.470922 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw_odh-ai-gateway-infra(b574a01d-cf97-4f79-887e-67665e66c7d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw" podUID="b574a01d-cf97-4f79-887e-67665e66c7d4" Jul 17 18:30:23.224274 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:23.224231 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b43b22bb-c888-443f-915b-fafbb3fe96f6" path="/var/lib/kubelet/pods/b43b22bb-c888-443f-915b-fafbb3fe96f6/volumes" Jul 17 18:30:23.495447 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:23.495365 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw"] Jul 17 18:30:23.624989 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:23.624967 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw" Jul 17 18:30:23.757968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:23.757889 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b574a01d-cf97-4f79-887e-67665e66c7d4-maas-api-service-ca\") pod \"b574a01d-cf97-4f79-887e-67665e66c7d4\" (UID: \"b574a01d-cf97-4f79-887e-67665e66c7d4\") " Jul 17 18:30:23.758261 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:23.758241 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b574a01d-cf97-4f79-887e-67665e66c7d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b574a01d-cf97-4f79-887e-67665e66c7d4" (UID: "b574a01d-cf97-4f79-887e-67665e66c7d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:30:23.859361 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:23.859330 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b574a01d-cf97-4f79-887e-67665e66c7d4-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:30:24.481399 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:24.481374 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw" Jul 17 18:30:24.481399 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:24.481395 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw" event={"ID":"b574a01d-cf97-4f79-887e-67665e66c7d4","Type":"ContainerDied","Data":"0ab792ec4ff8b77b23a9bd4ed0e0861f67c95b74b2556a678ffcd0d9532867f7"} Jul 17 18:30:24.481918 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:24.481449 2572 scope.go:117] "RemoveContainer" containerID="4369df6b031a046d2e0252c98339a4f5c8d06b24e4b2b363a59a1cb148018609" Jul 17 18:30:24.501055 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:24.501027 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw"] Jul 17 18:30:24.504610 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:24.504591 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-v5ssw"] Jul 17 18:30:25.225694 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:25.225662 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b574a01d-cf97-4f79-887e-67665e66c7d4" path="/var/lib/kubelet/pods/b574a01d-cf97-4f79-887e-67665e66c7d4/volumes" Jul 17 18:30:54.717630 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.717596 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww"] Jul 17 18:30:54.718113 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.717880 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b43b22bb-c888-443f-915b-fafbb3fe96f6" containerName="cleanup" Jul 17 18:30:54.718113 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.717890 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b43b22bb-c888-443f-915b-fafbb3fe96f6" containerName="cleanup" Jul 17 18:30:54.718113 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.717907 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b43b22bb-c888-443f-915b-fafbb3fe96f6" containerName="cleanup" Jul 17 18:30:54.718113 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.717913 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b43b22bb-c888-443f-915b-fafbb3fe96f6" containerName="cleanup" Jul 17 18:30:54.718113 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.717919 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b574a01d-cf97-4f79-887e-67665e66c7d4" containerName="revoke-keys" Jul 17 18:30:54.718113 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.717925 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b574a01d-cf97-4f79-887e-67665e66c7d4" containerName="revoke-keys" Jul 17 18:30:54.718113 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.717931 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b574a01d-cf97-4f79-887e-67665e66c7d4" containerName="revoke-keys" Jul 17 18:30:54.718113 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.717936 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b574a01d-cf97-4f79-887e-67665e66c7d4" containerName="revoke-keys" Jul 17 18:30:54.718113 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.717941 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b574a01d-cf97-4f79-887e-67665e66c7d4" containerName="revoke-keys" Jul 17 18:30:54.718113 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.717946 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b574a01d-cf97-4f79-887e-67665e66c7d4" containerName="revoke-keys" Jul 17 18:30:54.718113 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.717954 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b43b22bb-c888-443f-915b-fafbb3fe96f6" containerName="cleanup" Jul 17 18:30:54.718113 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.717959 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b43b22bb-c888-443f-915b-fafbb3fe96f6" containerName="cleanup" Jul 17 18:30:54.718113 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.718004 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b43b22bb-c888-443f-915b-fafbb3fe96f6" containerName="cleanup" Jul 17 18:30:54.718113 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.718026 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b574a01d-cf97-4f79-887e-67665e66c7d4" containerName="revoke-keys" Jul 17 18:30:54.718113 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.718033 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b574a01d-cf97-4f79-887e-67665e66c7d4" containerName="revoke-keys" Jul 17 18:30:54.718113 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.718039 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b43b22bb-c888-443f-915b-fafbb3fe96f6" containerName="cleanup" Jul 17 18:30:54.720885 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.720870 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww" Jul 17 18:30:54.723220 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.723197 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:30:54.723358 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.723237 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:30:54.729884 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.729861 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww"] Jul 17 18:30:54.803703 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.803673 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf872f8-d524-4be4-b607-267d6eb085d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww\" (UID: \"caf872f8-d524-4be4-b607-267d6eb085d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww" Jul 17 18:30:54.904739 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.904712 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf872f8-d524-4be4-b607-267d6eb085d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww\" (UID: \"caf872f8-d524-4be4-b607-267d6eb085d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww" Jul 17 18:30:54.905333 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:54.905316 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf872f8-d524-4be4-b607-267d6eb085d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww\" (UID: \"caf872f8-d524-4be4-b607-267d6eb085d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww" Jul 17 18:30:55.031625 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:55.031593 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww" Jul 17 18:30:55.157214 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:55.157190 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww"] Jul 17 18:30:55.159457 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:30:55.159429 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf872f8_d524_4be4_b607_267d6eb085d4.slice/crio-8fe7d928c5c9fcde2622b82d1995d94434f7d28ada10f45706012f6af8633848 WatchSource:0}: Error finding container 8fe7d928c5c9fcde2622b82d1995d94434f7d28ada10f45706012f6af8633848: Status 404 returned error can't find the container with id 8fe7d928c5c9fcde2622b82d1995d94434f7d28ada10f45706012f6af8633848 Jul 17 18:30:55.614459 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:55.614367 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww" event={"ID":"caf872f8-d524-4be4-b607-267d6eb085d4","Type":"ContainerStarted","Data":"cb7473fd51f20f5fcc7bf31cded81456bdafa6fafcc25991a0539be9fcf74f7e"} Jul 17 18:30:55.614459 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:55.614415 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww" event={"ID":"caf872f8-d524-4be4-b607-267d6eb085d4","Type":"ContainerStarted","Data":"8fe7d928c5c9fcde2622b82d1995d94434f7d28ada10f45706012f6af8633848"} Jul 17 18:30:55.630645 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:55.630597 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww" podStartSLOduration=1.630580154 podStartE2EDuration="1.630580154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:30:55.628159592 +0000 UTC m=+2051.037587220" watchObservedRunningTime="2026-07-17 18:30:55.630580154 +0000 UTC m=+2051.040007759" Jul 17 18:30:56.619768 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:56.619731 2572 generic.go:356] "Generic (PLEG): container finished" podID="caf872f8-d524-4be4-b607-267d6eb085d4" containerID="cb7473fd51f20f5fcc7bf31cded81456bdafa6fafcc25991a0539be9fcf74f7e" exitCode=6 Jul 17 18:30:56.619768 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:56.619772 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww" event={"ID":"caf872f8-d524-4be4-b607-267d6eb085d4","Type":"ContainerDied","Data":"cb7473fd51f20f5fcc7bf31cded81456bdafa6fafcc25991a0539be9fcf74f7e"} Jul 17 18:30:56.620192 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:56.619983 2572 scope.go:117] "RemoveContainer" containerID="cb7473fd51f20f5fcc7bf31cded81456bdafa6fafcc25991a0539be9fcf74f7e" Jul 17 18:30:56.870772 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:56.870719 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg"] Jul 17 18:30:56.871102 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:56.871090 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b43b22bb-c888-443f-915b-fafbb3fe96f6" containerName="cleanup" Jul 17 18:30:56.871147 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:56.871105 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b574a01d-cf97-4f79-887e-67665e66c7d4" containerName="revoke-keys" Jul 17 18:30:56.873823 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:56.873809 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg" Jul 17 18:30:56.881068 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:56.880992 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg"] Jul 17 18:30:56.919572 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:56.919538 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d00fcf-e333-4e96-b2f6-1229311d5542-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg\" (UID: \"f5d00fcf-e333-4e96-b2f6-1229311d5542\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg" Jul 17 18:30:57.020734 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:57.020693 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d00fcf-e333-4e96-b2f6-1229311d5542-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg\" (UID: \"f5d00fcf-e333-4e96-b2f6-1229311d5542\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg" Jul 17 18:30:57.021360 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:57.021341 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d00fcf-e333-4e96-b2f6-1229311d5542-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg\" (UID: \"f5d00fcf-e333-4e96-b2f6-1229311d5542\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg" Jul 17 18:30:57.184847 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:57.184749 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg" Jul 17 18:30:57.518573 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:57.518470 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg"] Jul 17 18:30:57.520795 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:30:57.520768 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d00fcf_e333_4e96_b2f6_1229311d5542.slice/crio-ca3e846581683e5e1453b42a9b30549a99bf2234304b2c3b281788bd0e6677d6 WatchSource:0}: Error finding container ca3e846581683e5e1453b42a9b30549a99bf2234304b2c3b281788bd0e6677d6: Status 404 returned error can't find the container with id ca3e846581683e5e1453b42a9b30549a99bf2234304b2c3b281788bd0e6677d6 Jul 17 18:30:57.625738 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:57.625705 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg" event={"ID":"f5d00fcf-e333-4e96-b2f6-1229311d5542","Type":"ContainerStarted","Data":"ca3e846581683e5e1453b42a9b30549a99bf2234304b2c3b281788bd0e6677d6"} Jul 17 18:30:57.627458 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:57.627430 2572 generic.go:356] "Generic (PLEG): container finished" podID="caf872f8-d524-4be4-b607-267d6eb085d4" containerID="606c7889b802bdf5f00f4c7b72bc9df88d54319c7d76b41ea48ac158aea925ad" exitCode=6 Jul 17 18:30:57.627584 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:57.627488 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww" event={"ID":"caf872f8-d524-4be4-b607-267d6eb085d4","Type":"ContainerDied","Data":"606c7889b802bdf5f00f4c7b72bc9df88d54319c7d76b41ea48ac158aea925ad"} Jul 17 18:30:57.627584 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:57.627538 2572 scope.go:117] "RemoveContainer" containerID="cb7473fd51f20f5fcc7bf31cded81456bdafa6fafcc25991a0539be9fcf74f7e" Jul 17 18:30:57.627753 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:57.627738 2572 scope.go:117] "RemoveContainer" containerID="606c7889b802bdf5f00f4c7b72bc9df88d54319c7d76b41ea48ac158aea925ad" Jul 17 18:30:57.627947 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:30:57.627930 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww_odh-ai-gateway-infra(caf872f8-d524-4be4-b607-267d6eb085d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww" podUID="caf872f8-d524-4be4-b607-267d6eb085d4" Jul 17 18:30:58.632820 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:58.632784 2572 generic.go:356] "Generic (PLEG): container finished" podID="f5d00fcf-e333-4e96-b2f6-1229311d5542" containerID="d1470db147a466e2640e18d9b4898f990ddc5767f291748e53657d80eed4c0bc" exitCode=6 Jul 17 18:30:58.633278 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:58.632875 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg" event={"ID":"f5d00fcf-e333-4e96-b2f6-1229311d5542","Type":"ContainerDied","Data":"d1470db147a466e2640e18d9b4898f990ddc5767f291748e53657d80eed4c0bc"} Jul 17 18:30:58.633278 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:58.633095 2572 scope.go:117] "RemoveContainer" containerID="d1470db147a466e2640e18d9b4898f990ddc5767f291748e53657d80eed4c0bc" Jul 17 18:30:58.634587 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:58.634568 2572 scope.go:117] "RemoveContainer" containerID="606c7889b802bdf5f00f4c7b72bc9df88d54319c7d76b41ea48ac158aea925ad" Jul 17 18:30:58.634706 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:30:58.634690 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww_odh-ai-gateway-infra(caf872f8-d524-4be4-b607-267d6eb085d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww" podUID="caf872f8-d524-4be4-b607-267d6eb085d4" Jul 17 18:30:59.640955 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:59.640921 2572 generic.go:356] "Generic (PLEG): container finished" podID="f5d00fcf-e333-4e96-b2f6-1229311d5542" containerID="704f9d781b25395db3b08d8e9262d0288979bcb95c8b31472a3640c5d9d7e429" exitCode=6 Jul 17 18:30:59.641412 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:59.641000 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg" event={"ID":"f5d00fcf-e333-4e96-b2f6-1229311d5542","Type":"ContainerDied","Data":"704f9d781b25395db3b08d8e9262d0288979bcb95c8b31472a3640c5d9d7e429"} Jul 17 18:30:59.641412 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:59.641086 2572 scope.go:117] "RemoveContainer" containerID="d1470db147a466e2640e18d9b4898f990ddc5767f291748e53657d80eed4c0bc" Jul 17 18:30:59.641412 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:30:59.641274 2572 scope.go:117] "RemoveContainer" containerID="704f9d781b25395db3b08d8e9262d0288979bcb95c8b31472a3640c5d9d7e429" Jul 17 18:30:59.641551 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:30:59.641526 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg_odh-ai-gateway-infra(f5d00fcf-e333-4e96-b2f6-1229311d5542)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg" podUID="f5d00fcf-e333-4e96-b2f6-1229311d5542" Jul 17 18:31:00.646732 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:00.646703 2572 scope.go:117] "RemoveContainer" containerID="704f9d781b25395db3b08d8e9262d0288979bcb95c8b31472a3640c5d9d7e429" Jul 17 18:31:00.647141 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:31:00.646903 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg_odh-ai-gateway-infra(f5d00fcf-e333-4e96-b2f6-1229311d5542)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg" podUID="f5d00fcf-e333-4e96-b2f6-1229311d5542" Jul 17 18:31:12.219434 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:12.219346 2572 scope.go:117] "RemoveContainer" containerID="606c7889b802bdf5f00f4c7b72bc9df88d54319c7d76b41ea48ac158aea925ad" Jul 17 18:31:12.698197 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:12.698162 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww" event={"ID":"caf872f8-d524-4be4-b607-267d6eb085d4","Type":"ContainerStarted","Data":"391a3ac8f9faf94062f63003297f91c8c2545dbb5f0f25001dd79bf29530e906"} Jul 17 18:31:13.704108 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:13.704073 2572 generic.go:356] "Generic (PLEG): container finished" podID="caf872f8-d524-4be4-b607-267d6eb085d4" containerID="391a3ac8f9faf94062f63003297f91c8c2545dbb5f0f25001dd79bf29530e906" exitCode=6 Jul 17 18:31:13.704586 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:13.704131 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww" event={"ID":"caf872f8-d524-4be4-b607-267d6eb085d4","Type":"ContainerDied","Data":"391a3ac8f9faf94062f63003297f91c8c2545dbb5f0f25001dd79bf29530e906"} Jul 17 18:31:13.704586 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:13.704178 2572 scope.go:117] "RemoveContainer" containerID="606c7889b802bdf5f00f4c7b72bc9df88d54319c7d76b41ea48ac158aea925ad" Jul 17 18:31:13.704586 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:13.704488 2572 scope.go:117] "RemoveContainer" containerID="391a3ac8f9faf94062f63003297f91c8c2545dbb5f0f25001dd79bf29530e906" Jul 17 18:31:13.704731 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:31:13.704714 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww_odh-ai-gateway-infra(caf872f8-d524-4be4-b607-267d6eb085d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww" podUID="caf872f8-d524-4be4-b607-267d6eb085d4" Jul 17 18:31:13.725522 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:13.725497 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww"] Jul 17 18:31:14.219976 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:14.219950 2572 scope.go:117] "RemoveContainer" containerID="704f9d781b25395db3b08d8e9262d0288979bcb95c8b31472a3640c5d9d7e429" Jul 17 18:31:14.710386 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:14.710350 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg" event={"ID":"f5d00fcf-e333-4e96-b2f6-1229311d5542","Type":"ContainerStarted","Data":"6fde6f28cd15c1982de238c5ced5093aea79511dd02319a89b047c68a79a4edf"} Jul 17 18:31:14.731037 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:14.729637 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg" podStartSLOduration=18.729616975 podStartE2EDuration="18.729616975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:31:14.724502535 +0000 UTC m=+2070.133930141" watchObservedRunningTime="2026-07-17 18:31:14.729616975 +0000 UTC m=+2070.139044581" Jul 17 18:31:14.847496 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:14.847470 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww" Jul 17 18:31:14.975826 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:14.975798 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf872f8-d524-4be4-b607-267d6eb085d4-maas-api-service-ca\") pod \"caf872f8-d524-4be4-b607-267d6eb085d4\" (UID: \"caf872f8-d524-4be4-b607-267d6eb085d4\") " Jul 17 18:31:14.976172 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:14.976145 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf872f8-d524-4be4-b607-267d6eb085d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "caf872f8-d524-4be4-b607-267d6eb085d4" (UID: "caf872f8-d524-4be4-b607-267d6eb085d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:31:15.076647 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:15.076614 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf872f8-d524-4be4-b607-267d6eb085d4-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:31:15.716761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:15.716733 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww" Jul 17 18:31:15.717232 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:15.716772 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww" event={"ID":"caf872f8-d524-4be4-b607-267d6eb085d4","Type":"ContainerDied","Data":"8fe7d928c5c9fcde2622b82d1995d94434f7d28ada10f45706012f6af8633848"} Jul 17 18:31:15.717232 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:15.716832 2572 scope.go:117] "RemoveContainer" containerID="391a3ac8f9faf94062f63003297f91c8c2545dbb5f0f25001dd79bf29530e906" Jul 17 18:31:15.718759 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:15.718711 2572 generic.go:356] "Generic (PLEG): container finished" podID="f5d00fcf-e333-4e96-b2f6-1229311d5542" containerID="6fde6f28cd15c1982de238c5ced5093aea79511dd02319a89b047c68a79a4edf" exitCode=6 Jul 17 18:31:15.718935 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:15.718905 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg" event={"ID":"f5d00fcf-e333-4e96-b2f6-1229311d5542","Type":"ContainerDied","Data":"6fde6f28cd15c1982de238c5ced5093aea79511dd02319a89b047c68a79a4edf"} Jul 17 18:31:15.719320 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:15.719294 2572 scope.go:117] "RemoveContainer" containerID="6fde6f28cd15c1982de238c5ced5093aea79511dd02319a89b047c68a79a4edf" Jul 17 18:31:15.719540 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:31:15.719519 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg_odh-ai-gateway-infra(f5d00fcf-e333-4e96-b2f6-1229311d5542)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg" podUID="f5d00fcf-e333-4e96-b2f6-1229311d5542" Jul 17 18:31:15.727489 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:15.727472 2572 scope.go:117] "RemoveContainer" containerID="704f9d781b25395db3b08d8e9262d0288979bcb95c8b31472a3640c5d9d7e429" Jul 17 18:31:15.734087 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:15.734002 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg"] Jul 17 18:31:15.738179 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:15.738157 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww"] Jul 17 18:31:15.744507 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:15.744487 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-fv2ww"] Jul 17 18:31:16.850507 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:16.850483 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg" Jul 17 18:31:16.992586 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:16.992510 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d00fcf-e333-4e96-b2f6-1229311d5542-maas-api-service-ca\") pod \"f5d00fcf-e333-4e96-b2f6-1229311d5542\" (UID: \"f5d00fcf-e333-4e96-b2f6-1229311d5542\") " Jul 17 18:31:16.992848 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:16.992828 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d00fcf-e333-4e96-b2f6-1229311d5542-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5d00fcf-e333-4e96-b2f6-1229311d5542" (UID: "f5d00fcf-e333-4e96-b2f6-1229311d5542"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:31:17.093227 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:17.093195 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d00fcf-e333-4e96-b2f6-1229311d5542-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:31:17.223911 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:17.223877 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caf872f8-d524-4be4-b607-267d6eb085d4" path="/var/lib/kubelet/pods/caf872f8-d524-4be4-b607-267d6eb085d4/volumes" Jul 17 18:31:17.730640 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:17.730603 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg" event={"ID":"f5d00fcf-e333-4e96-b2f6-1229311d5542","Type":"ContainerDied","Data":"ca3e846581683e5e1453b42a9b30549a99bf2234304b2c3b281788bd0e6677d6"} Jul 17 18:31:17.730640 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:17.730633 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg" Jul 17 18:31:17.730862 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:17.730663 2572 scope.go:117] "RemoveContainer" containerID="6fde6f28cd15c1982de238c5ced5093aea79511dd02319a89b047c68a79a4edf" Jul 17 18:31:17.746780 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:17.746756 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg"] Jul 17 18:31:17.753230 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:17.753207 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-p62zg"] Jul 17 18:31:19.224855 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:19.224827 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5d00fcf-e333-4e96-b2f6-1229311d5542" path="/var/lib/kubelet/pods/f5d00fcf-e333-4e96-b2f6-1229311d5542/volumes" Jul 17 18:31:54.770910 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.770877 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp"] Jul 17 18:31:54.771350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.771204 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf872f8-d524-4be4-b607-267d6eb085d4" containerName="revoke-keys" Jul 17 18:31:54.771350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.771216 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf872f8-d524-4be4-b607-267d6eb085d4" containerName="revoke-keys" Jul 17 18:31:54.771350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.771230 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d00fcf-e333-4e96-b2f6-1229311d5542" containerName="revoke-keys" Jul 17 18:31:54.771350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.771235 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d00fcf-e333-4e96-b2f6-1229311d5542" containerName="revoke-keys" Jul 17 18:31:54.771350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.771242 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d00fcf-e333-4e96-b2f6-1229311d5542" containerName="revoke-keys" Jul 17 18:31:54.771350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.771247 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d00fcf-e333-4e96-b2f6-1229311d5542" containerName="revoke-keys" Jul 17 18:31:54.771350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.771258 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf872f8-d524-4be4-b607-267d6eb085d4" containerName="revoke-keys" Jul 17 18:31:54.771350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.771263 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf872f8-d524-4be4-b607-267d6eb085d4" containerName="revoke-keys" Jul 17 18:31:54.771350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.771309 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5d00fcf-e333-4e96-b2f6-1229311d5542" containerName="revoke-keys" Jul 17 18:31:54.771350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.771317 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="caf872f8-d524-4be4-b607-267d6eb085d4" containerName="revoke-keys" Jul 17 18:31:54.771350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.771325 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="caf872f8-d524-4be4-b607-267d6eb085d4" containerName="revoke-keys" Jul 17 18:31:54.771350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.771330 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5d00fcf-e333-4e96-b2f6-1229311d5542" containerName="revoke-keys" Jul 17 18:31:54.774191 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.774176 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp" Jul 17 18:31:54.776970 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.776951 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:31:54.777107 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.776977 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:31:54.781618 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.781594 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp"] Jul 17 18:31:54.806025 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.805992 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c02f808-f405-4655-9763-5fd45f846f63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp\" (UID: \"5c02f808-f405-4655-9763-5fd45f846f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp" Jul 17 18:31:54.906501 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.906476 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c02f808-f405-4655-9763-5fd45f846f63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp\" (UID: \"5c02f808-f405-4655-9763-5fd45f846f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp" Jul 17 18:31:54.907104 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:54.907085 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c02f808-f405-4655-9763-5fd45f846f63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp\" (UID: \"5c02f808-f405-4655-9763-5fd45f846f63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp" Jul 17 18:31:55.085047 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:55.085019 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp" Jul 17 18:31:55.211913 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:55.211888 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp"] Jul 17 18:31:55.214508 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:31:55.214481 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c02f808_f405_4655_9763_5fd45f846f63.slice/crio-c489cb6fa2e140779e7d6ea44ae1baba68cb6ed0ae1b59560817bbd3a210496f WatchSource:0}: Error finding container c489cb6fa2e140779e7d6ea44ae1baba68cb6ed0ae1b59560817bbd3a210496f: Status 404 returned error can't find the container with id c489cb6fa2e140779e7d6ea44ae1baba68cb6ed0ae1b59560817bbd3a210496f Jul 17 18:31:55.885274 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:55.885239 2572 generic.go:356] "Generic (PLEG): container finished" podID="5c02f808-f405-4655-9763-5fd45f846f63" containerID="3f216c761665450216f29edd0b639e6bac461ace8db8ce9e75c31e2b7d43d60f" exitCode=6 Jul 17 18:31:55.885584 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:55.885304 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp" event={"ID":"5c02f808-f405-4655-9763-5fd45f846f63","Type":"ContainerDied","Data":"3f216c761665450216f29edd0b639e6bac461ace8db8ce9e75c31e2b7d43d60f"} Jul 17 18:31:55.885584 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:55.885343 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp" event={"ID":"5c02f808-f405-4655-9763-5fd45f846f63","Type":"ContainerStarted","Data":"c489cb6fa2e140779e7d6ea44ae1baba68cb6ed0ae1b59560817bbd3a210496f"} Jul 17 18:31:55.885584 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:55.885477 2572 scope.go:117] "RemoveContainer" containerID="3f216c761665450216f29edd0b639e6bac461ace8db8ce9e75c31e2b7d43d60f" Jul 17 18:31:56.891142 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:56.891106 2572 generic.go:356] "Generic (PLEG): container finished" podID="5c02f808-f405-4655-9763-5fd45f846f63" containerID="6916a63df24f02d7c33044fe83e6839a58fb20ba13f5097b48c7a969e6e00cd1" exitCode=6 Jul 17 18:31:56.891592 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:56.891196 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp" event={"ID":"5c02f808-f405-4655-9763-5fd45f846f63","Type":"ContainerDied","Data":"6916a63df24f02d7c33044fe83e6839a58fb20ba13f5097b48c7a969e6e00cd1"} Jul 17 18:31:56.891592 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:56.891274 2572 scope.go:117] "RemoveContainer" containerID="3f216c761665450216f29edd0b639e6bac461ace8db8ce9e75c31e2b7d43d60f" Jul 17 18:31:56.891592 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:56.891481 2572 scope.go:117] "RemoveContainer" containerID="6916a63df24f02d7c33044fe83e6839a58fb20ba13f5097b48c7a969e6e00cd1" Jul 17 18:31:56.891723 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:31:56.891694 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp_odh-ai-gateway-infra(5c02f808-f405-4655-9763-5fd45f846f63)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp" podUID="5c02f808-f405-4655-9763-5fd45f846f63" Jul 17 18:31:56.916031 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:56.915997 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g"] Jul 17 18:31:56.916355 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:56.916339 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf872f8-d524-4be4-b607-267d6eb085d4" containerName="revoke-keys" Jul 17 18:31:56.916437 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:56.916358 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf872f8-d524-4be4-b607-267d6eb085d4" containerName="revoke-keys" Jul 17 18:31:56.916437 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:56.916400 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d00fcf-e333-4e96-b2f6-1229311d5542" containerName="revoke-keys" Jul 17 18:31:56.916437 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:56.916409 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d00fcf-e333-4e96-b2f6-1229311d5542" containerName="revoke-keys" Jul 17 18:31:56.916579 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:56.916483 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="caf872f8-d524-4be4-b607-267d6eb085d4" containerName="revoke-keys" Jul 17 18:31:56.916579 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:56.916501 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5d00fcf-e333-4e96-b2f6-1229311d5542" containerName="revoke-keys" Jul 17 18:31:56.920304 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:56.920287 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g" Jul 17 18:31:56.925710 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:56.925687 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g"] Jul 17 18:31:57.024765 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:57.024734 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5502e584-0acc-4d41-b075-2eb8795507ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g\" (UID: \"5502e584-0acc-4d41-b075-2eb8795507ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g" Jul 17 18:31:57.125165 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:57.125132 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5502e584-0acc-4d41-b075-2eb8795507ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g\" (UID: \"5502e584-0acc-4d41-b075-2eb8795507ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g" Jul 17 18:31:57.125734 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:57.125715 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5502e584-0acc-4d41-b075-2eb8795507ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g\" (UID: \"5502e584-0acc-4d41-b075-2eb8795507ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g" Jul 17 18:31:57.231526 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:57.231456 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g" Jul 17 18:31:57.354938 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:57.354845 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g"] Jul 17 18:31:57.356973 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:31:57.356946 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5502e584_0acc_4d41_b075_2eb8795507ef.slice/crio-24a2c2598cc1d0d0ec54ff0180d53e92a8e0065158378939b1e7e651ebeca0fe WatchSource:0}: Error finding container 24a2c2598cc1d0d0ec54ff0180d53e92a8e0065158378939b1e7e651ebeca0fe: Status 404 returned error can't find the container with id 24a2c2598cc1d0d0ec54ff0180d53e92a8e0065158378939b1e7e651ebeca0fe Jul 17 18:31:57.897002 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:57.896920 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g" event={"ID":"5502e584-0acc-4d41-b075-2eb8795507ef","Type":"ContainerStarted","Data":"98a282f385a4ed7357591e7c8c94d0cbfaf3eecfa3ac07e826e187fbcf0a80f3"} Jul 17 18:31:57.897002 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:57.896983 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g" event={"ID":"5502e584-0acc-4d41-b075-2eb8795507ef","Type":"ContainerStarted","Data":"24a2c2598cc1d0d0ec54ff0180d53e92a8e0065158378939b1e7e651ebeca0fe"} Jul 17 18:31:57.898732 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:57.898713 2572 scope.go:117] "RemoveContainer" containerID="6916a63df24f02d7c33044fe83e6839a58fb20ba13f5097b48c7a969e6e00cd1" Jul 17 18:31:57.898892 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:31:57.898876 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp_odh-ai-gateway-infra(5c02f808-f405-4655-9763-5fd45f846f63)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp" podUID="5c02f808-f405-4655-9763-5fd45f846f63" Jul 17 18:31:57.911729 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:57.911684 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g" podStartSLOduration=1.911670513 podStartE2EDuration="1.911670513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:31:57.910496281 +0000 UTC m=+2113.319923908" watchObservedRunningTime="2026-07-17 18:31:57.911670513 +0000 UTC m=+2113.321098166" Jul 17 18:31:58.903640 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:58.903604 2572 generic.go:356] "Generic (PLEG): container finished" podID="5502e584-0acc-4d41-b075-2eb8795507ef" containerID="98a282f385a4ed7357591e7c8c94d0cbfaf3eecfa3ac07e826e187fbcf0a80f3" exitCode=6 Jul 17 18:31:58.904110 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:58.903691 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g" event={"ID":"5502e584-0acc-4d41-b075-2eb8795507ef","Type":"ContainerDied","Data":"98a282f385a4ed7357591e7c8c94d0cbfaf3eecfa3ac07e826e187fbcf0a80f3"} Jul 17 18:31:58.904110 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:58.903882 2572 scope.go:117] "RemoveContainer" containerID="98a282f385a4ed7357591e7c8c94d0cbfaf3eecfa3ac07e826e187fbcf0a80f3" Jul 17 18:31:59.909684 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:59.909589 2572 generic.go:356] "Generic (PLEG): container finished" podID="5502e584-0acc-4d41-b075-2eb8795507ef" containerID="bd56f1e4bea445618b897ed1170d70ec94891f7d84d4d4aacfcb5b7ee63eab80" exitCode=6 Jul 17 18:31:59.909684 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:59.909645 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g" event={"ID":"5502e584-0acc-4d41-b075-2eb8795507ef","Type":"ContainerDied","Data":"bd56f1e4bea445618b897ed1170d70ec94891f7d84d4d4aacfcb5b7ee63eab80"} Jul 17 18:31:59.910187 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:59.909734 2572 scope.go:117] "RemoveContainer" containerID="98a282f385a4ed7357591e7c8c94d0cbfaf3eecfa3ac07e826e187fbcf0a80f3" Jul 17 18:31:59.910187 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:31:59.909896 2572 scope.go:117] "RemoveContainer" containerID="bd56f1e4bea445618b897ed1170d70ec94891f7d84d4d4aacfcb5b7ee63eab80" Jul 17 18:31:59.910187 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:31:59.910148 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g_odh-ai-gateway-infra(5502e584-0acc-4d41-b075-2eb8795507ef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g" podUID="5502e584-0acc-4d41-b075-2eb8795507ef" Jul 17 18:32:00.915476 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:00.915443 2572 scope.go:117] "RemoveContainer" containerID="bd56f1e4bea445618b897ed1170d70ec94891f7d84d4d4aacfcb5b7ee63eab80" Jul 17 18:32:00.915872 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:32:00.915609 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g_odh-ai-gateway-infra(5502e584-0acc-4d41-b075-2eb8795507ef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g" podUID="5502e584-0acc-4d41-b075-2eb8795507ef" Jul 17 18:32:09.220093 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:09.220065 2572 scope.go:117] "RemoveContainer" containerID="6916a63df24f02d7c33044fe83e6839a58fb20ba13f5097b48c7a969e6e00cd1" Jul 17 18:32:09.952598 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:09.952564 2572 generic.go:356] "Generic (PLEG): container finished" podID="5c02f808-f405-4655-9763-5fd45f846f63" containerID="e7242c9268de4ce3c73e98661731b96960f9905f9908ae6ade01d830caa96945" exitCode=6 Jul 17 18:32:09.952725 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:09.952633 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp" event={"ID":"5c02f808-f405-4655-9763-5fd45f846f63","Type":"ContainerDied","Data":"e7242c9268de4ce3c73e98661731b96960f9905f9908ae6ade01d830caa96945"} Jul 17 18:32:09.952725 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:09.952690 2572 scope.go:117] "RemoveContainer" containerID="6916a63df24f02d7c33044fe83e6839a58fb20ba13f5097b48c7a969e6e00cd1" Jul 17 18:32:09.952952 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:09.952926 2572 scope.go:117] "RemoveContainer" containerID="e7242c9268de4ce3c73e98661731b96960f9905f9908ae6ade01d830caa96945" Jul 17 18:32:09.953203 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:32:09.953183 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp_odh-ai-gateway-infra(5c02f808-f405-4655-9763-5fd45f846f63)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp" podUID="5c02f808-f405-4655-9763-5fd45f846f63" Jul 17 18:32:10.978181 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:10.978146 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp"] Jul 17 18:32:11.110058 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:11.110034 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp" Jul 17 18:32:11.244137 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:11.244062 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c02f808-f405-4655-9763-5fd45f846f63-maas-api-service-ca\") pod \"5c02f808-f405-4655-9763-5fd45f846f63\" (UID: \"5c02f808-f405-4655-9763-5fd45f846f63\") " Jul 17 18:32:11.244439 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:11.244418 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c02f808-f405-4655-9763-5fd45f846f63-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5c02f808-f405-4655-9763-5fd45f846f63" (UID: "5c02f808-f405-4655-9763-5fd45f846f63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:32:11.345098 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:11.345068 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c02f808-f405-4655-9763-5fd45f846f63-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:32:11.963239 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:11.963208 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp" Jul 17 18:32:11.963411 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:11.963202 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp" event={"ID":"5c02f808-f405-4655-9763-5fd45f846f63","Type":"ContainerDied","Data":"c489cb6fa2e140779e7d6ea44ae1baba68cb6ed0ae1b59560817bbd3a210496f"} Jul 17 18:32:11.963411 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:11.963357 2572 scope.go:117] "RemoveContainer" containerID="e7242c9268de4ce3c73e98661731b96960f9905f9908ae6ade01d830caa96945" Jul 17 18:32:11.983418 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:11.983396 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp"] Jul 17 18:32:11.988820 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:11.988798 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hqwkp"] Jul 17 18:32:13.224463 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:13.224431 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c02f808-f405-4655-9763-5fd45f846f63" path="/var/lib/kubelet/pods/5c02f808-f405-4655-9763-5fd45f846f63/volumes" Jul 17 18:32:14.219787 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:14.219757 2572 scope.go:117] "RemoveContainer" containerID="bd56f1e4bea445618b897ed1170d70ec94891f7d84d4d4aacfcb5b7ee63eab80" Jul 17 18:32:14.977759 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:14.977729 2572 generic.go:356] "Generic (PLEG): container finished" podID="5502e584-0acc-4d41-b075-2eb8795507ef" containerID="3e3a8fc6d8413d0efb514a1cd0c9267f74af8555bff522c455016d76c4d19dad" exitCode=6 Jul 17 18:32:14.978176 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:14.977776 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g" event={"ID":"5502e584-0acc-4d41-b075-2eb8795507ef","Type":"ContainerDied","Data":"3e3a8fc6d8413d0efb514a1cd0c9267f74af8555bff522c455016d76c4d19dad"} Jul 17 18:32:14.978176 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:14.977823 2572 scope.go:117] "RemoveContainer" containerID="bd56f1e4bea445618b897ed1170d70ec94891f7d84d4d4aacfcb5b7ee63eab80" Jul 17 18:32:14.978176 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:14.978097 2572 scope.go:117] "RemoveContainer" containerID="3e3a8fc6d8413d0efb514a1cd0c9267f74af8555bff522c455016d76c4d19dad" Jul 17 18:32:14.978320 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:32:14.978297 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g_odh-ai-gateway-infra(5502e584-0acc-4d41-b075-2eb8795507ef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g" podUID="5502e584-0acc-4d41-b075-2eb8795507ef" Jul 17 18:32:16.002325 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:16.002295 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g"] Jul 17 18:32:16.137860 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:16.137839 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g" Jul 17 18:32:16.289348 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:16.289320 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5502e584-0acc-4d41-b075-2eb8795507ef-maas-api-service-ca\") pod \"5502e584-0acc-4d41-b075-2eb8795507ef\" (UID: \"5502e584-0acc-4d41-b075-2eb8795507ef\") " Jul 17 18:32:16.289655 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:16.289633 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5502e584-0acc-4d41-b075-2eb8795507ef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5502e584-0acc-4d41-b075-2eb8795507ef" (UID: "5502e584-0acc-4d41-b075-2eb8795507ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:32:16.390410 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:16.390378 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5502e584-0acc-4d41-b075-2eb8795507ef-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:32:16.989298 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:16.989260 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g" event={"ID":"5502e584-0acc-4d41-b075-2eb8795507ef","Type":"ContainerDied","Data":"24a2c2598cc1d0d0ec54ff0180d53e92a8e0065158378939b1e7e651ebeca0fe"} Jul 17 18:32:16.989298 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:16.989293 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g" Jul 17 18:32:16.989522 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:16.989333 2572 scope.go:117] "RemoveContainer" containerID="3e3a8fc6d8413d0efb514a1cd0c9267f74af8555bff522c455016d76c4d19dad" Jul 17 18:32:17.014511 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:17.014488 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g"] Jul 17 18:32:17.021144 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:17.021121 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-wwp9g"] Jul 17 18:32:17.224965 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:17.224931 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5502e584-0acc-4d41-b075-2eb8795507ef" path="/var/lib/kubelet/pods/5502e584-0acc-4d41-b075-2eb8795507ef/volumes" Jul 17 18:32:44.819392 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.819299 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98"] Jul 17 18:32:44.819929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.819606 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c02f808-f405-4655-9763-5fd45f846f63" containerName="revoke-keys" Jul 17 18:32:44.819929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.819617 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c02f808-f405-4655-9763-5fd45f846f63" containerName="revoke-keys" Jul 17 18:32:44.819929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.819632 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5502e584-0acc-4d41-b075-2eb8795507ef" containerName="revoke-keys" Jul 17 18:32:44.819929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.819637 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5502e584-0acc-4d41-b075-2eb8795507ef" containerName="revoke-keys" Jul 17 18:32:44.819929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.819643 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5502e584-0acc-4d41-b075-2eb8795507ef" containerName="revoke-keys" Jul 17 18:32:44.819929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.819649 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5502e584-0acc-4d41-b075-2eb8795507ef" containerName="revoke-keys" Jul 17 18:32:44.819929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.819657 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c02f808-f405-4655-9763-5fd45f846f63" containerName="revoke-keys" Jul 17 18:32:44.819929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.819663 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c02f808-f405-4655-9763-5fd45f846f63" containerName="revoke-keys" Jul 17 18:32:44.819929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.819669 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c02f808-f405-4655-9763-5fd45f846f63" containerName="revoke-keys" Jul 17 18:32:44.819929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.819673 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c02f808-f405-4655-9763-5fd45f846f63" containerName="revoke-keys" Jul 17 18:32:44.819929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.819683 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5502e584-0acc-4d41-b075-2eb8795507ef" containerName="revoke-keys" Jul 17 18:32:44.819929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.819689 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5502e584-0acc-4d41-b075-2eb8795507ef" containerName="revoke-keys" Jul 17 18:32:44.819929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.819733 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5502e584-0acc-4d41-b075-2eb8795507ef" containerName="revoke-keys" Jul 17 18:32:44.819929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.819742 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5502e584-0acc-4d41-b075-2eb8795507ef" containerName="revoke-keys" Jul 17 18:32:44.819929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.819749 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c02f808-f405-4655-9763-5fd45f846f63" containerName="revoke-keys" Jul 17 18:32:44.819929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.819755 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c02f808-f405-4655-9763-5fd45f846f63" containerName="revoke-keys" Jul 17 18:32:44.822763 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.822747 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98" Jul 17 18:32:44.825502 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.825481 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:32:44.825502 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.825501 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:32:44.829543 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.829510 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98"] Jul 17 18:32:44.928197 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:44.928158 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a74b088-084f-4566-b7ac-ad7409b10fc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98\" (UID: \"9a74b088-084f-4566-b7ac-ad7409b10fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98" Jul 17 18:32:45.028738 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:45.028692 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a74b088-084f-4566-b7ac-ad7409b10fc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98\" (UID: \"9a74b088-084f-4566-b7ac-ad7409b10fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98" Jul 17 18:32:45.029336 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:45.029317 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a74b088-084f-4566-b7ac-ad7409b10fc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98\" (UID: \"9a74b088-084f-4566-b7ac-ad7409b10fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98" Jul 17 18:32:45.137057 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:45.136966 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:32:45.144919 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:45.144899 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98" Jul 17 18:32:45.267153 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:45.267132 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98"] Jul 17 18:32:45.269093 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:32:45.269068 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a74b088_084f_4566_b7ac_ad7409b10fc4.slice/crio-d7bb9494ec0bc3b6f0cf6f5caabcf0d0294e4d0e6aad219c88a25624d902de06 WatchSource:0}: Error finding container d7bb9494ec0bc3b6f0cf6f5caabcf0d0294e4d0e6aad219c88a25624d902de06: Status 404 returned error can't find the container with id d7bb9494ec0bc3b6f0cf6f5caabcf0d0294e4d0e6aad219c88a25624d902de06 Jul 17 18:32:46.107937 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:46.107900 2572 generic.go:356] "Generic (PLEG): container finished" podID="9a74b088-084f-4566-b7ac-ad7409b10fc4" containerID="3b17647d5f2358ec8cca94164baed84b1ebb35804fb1a6701966b41e3ab90176" exitCode=6 Jul 17 18:32:46.108380 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:46.107984 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98" event={"ID":"9a74b088-084f-4566-b7ac-ad7409b10fc4","Type":"ContainerDied","Data":"3b17647d5f2358ec8cca94164baed84b1ebb35804fb1a6701966b41e3ab90176"} Jul 17 18:32:46.108380 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:46.108081 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98" event={"ID":"9a74b088-084f-4566-b7ac-ad7409b10fc4","Type":"ContainerStarted","Data":"d7bb9494ec0bc3b6f0cf6f5caabcf0d0294e4d0e6aad219c88a25624d902de06"} Jul 17 18:32:46.108380 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:46.108173 2572 scope.go:117] "RemoveContainer" containerID="3b17647d5f2358ec8cca94164baed84b1ebb35804fb1a6701966b41e3ab90176" Jul 17 18:32:47.114122 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:47.114089 2572 generic.go:356] "Generic (PLEG): container finished" podID="9a74b088-084f-4566-b7ac-ad7409b10fc4" containerID="131cd6f40a46b292ca5591dab1001efef2023ae9f9f06faa2c44d2d28203ba94" exitCode=6 Jul 17 18:32:47.114638 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:47.114178 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98" event={"ID":"9a74b088-084f-4566-b7ac-ad7409b10fc4","Type":"ContainerDied","Data":"131cd6f40a46b292ca5591dab1001efef2023ae9f9f06faa2c44d2d28203ba94"} Jul 17 18:32:47.114638 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:47.114240 2572 scope.go:117] "RemoveContainer" containerID="3b17647d5f2358ec8cca94164baed84b1ebb35804fb1a6701966b41e3ab90176" Jul 17 18:32:47.114638 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:47.114395 2572 scope.go:117] "RemoveContainer" containerID="131cd6f40a46b292ca5591dab1001efef2023ae9f9f06faa2c44d2d28203ba94" Jul 17 18:32:47.114638 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:32:47.114604 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98_odh-ai-gateway-infra(9a74b088-084f-4566-b7ac-ad7409b10fc4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98" podUID="9a74b088-084f-4566-b7ac-ad7409b10fc4" Jul 17 18:32:48.119449 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:48.119424 2572 scope.go:117] "RemoveContainer" containerID="131cd6f40a46b292ca5591dab1001efef2023ae9f9f06faa2c44d2d28203ba94" Jul 17 18:32:48.119820 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:32:48.119635 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98_odh-ai-gateway-infra(9a74b088-084f-4566-b7ac-ad7409b10fc4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98" podUID="9a74b088-084f-4566-b7ac-ad7409b10fc4" Jul 17 18:32:56.965515 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:56.965481 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266"] Jul 17 18:32:56.965924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:56.965814 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c02f808-f405-4655-9763-5fd45f846f63" containerName="revoke-keys" Jul 17 18:32:56.965924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:56.965828 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5502e584-0acc-4d41-b075-2eb8795507ef" containerName="revoke-keys" Jul 17 18:32:56.968706 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:56.968679 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266" Jul 17 18:32:56.977394 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:56.977371 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266"] Jul 17 18:32:57.025002 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:57.024967 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae99619c-70a8-498a-b99f-3197a162afac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-w9266\" (UID: \"ae99619c-70a8-498a-b99f-3197a162afac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266" Jul 17 18:32:57.125491 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:57.125457 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae99619c-70a8-498a-b99f-3197a162afac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-w9266\" (UID: \"ae99619c-70a8-498a-b99f-3197a162afac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266" Jul 17 18:32:57.126092 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:57.126075 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae99619c-70a8-498a-b99f-3197a162afac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-w9266\" (UID: \"ae99619c-70a8-498a-b99f-3197a162afac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266" Jul 17 18:32:57.279273 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:57.279245 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266" Jul 17 18:32:57.405933 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:57.405901 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266"] Jul 17 18:32:57.408881 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:32:57.408840 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae99619c_70a8_498a_b99f_3197a162afac.slice/crio-888d1f3845afe44b295aea61e0a99611eafb46972c6c0c52419f259f0fef6e99 WatchSource:0}: Error finding container 888d1f3845afe44b295aea61e0a99611eafb46972c6c0c52419f259f0fef6e99: Status 404 returned error can't find the container with id 888d1f3845afe44b295aea61e0a99611eafb46972c6c0c52419f259f0fef6e99 Jul 17 18:32:58.160176 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:58.160146 2572 generic.go:356] "Generic (PLEG): container finished" podID="ae99619c-70a8-498a-b99f-3197a162afac" containerID="c72b1e18e8ad9e6f33d16b47f64325efa8c636481dcd5a02cde33730c375bdfa" exitCode=6 Jul 17 18:32:58.160561 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:58.160235 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266" event={"ID":"ae99619c-70a8-498a-b99f-3197a162afac","Type":"ContainerDied","Data":"c72b1e18e8ad9e6f33d16b47f64325efa8c636481dcd5a02cde33730c375bdfa"} Jul 17 18:32:58.160561 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:58.160282 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266" event={"ID":"ae99619c-70a8-498a-b99f-3197a162afac","Type":"ContainerStarted","Data":"888d1f3845afe44b295aea61e0a99611eafb46972c6c0c52419f259f0fef6e99"} Jul 17 18:32:58.160561 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:58.160396 2572 scope.go:117] "RemoveContainer" containerID="c72b1e18e8ad9e6f33d16b47f64325efa8c636481dcd5a02cde33730c375bdfa" Jul 17 18:32:59.165957 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:59.165921 2572 generic.go:356] "Generic (PLEG): container finished" podID="ae99619c-70a8-498a-b99f-3197a162afac" containerID="7979e6ae1d9fbed67bc0374ea5a711cc4bd98f1b50d0a54b9b6dc8f30fc3f985" exitCode=6 Jul 17 18:32:59.166373 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:59.166027 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266" event={"ID":"ae99619c-70a8-498a-b99f-3197a162afac","Type":"ContainerDied","Data":"7979e6ae1d9fbed67bc0374ea5a711cc4bd98f1b50d0a54b9b6dc8f30fc3f985"} Jul 17 18:32:59.166373 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:59.166094 2572 scope.go:117] "RemoveContainer" containerID="c72b1e18e8ad9e6f33d16b47f64325efa8c636481dcd5a02cde33730c375bdfa" Jul 17 18:32:59.166373 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:32:59.166316 2572 scope.go:117] "RemoveContainer" containerID="7979e6ae1d9fbed67bc0374ea5a711cc4bd98f1b50d0a54b9b6dc8f30fc3f985" Jul 17 18:32:59.166556 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:32:59.166514 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-w9266_odh-ai-gateway-infra(ae99619c-70a8-498a-b99f-3197a162afac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266" podUID="ae99619c-70a8-498a-b99f-3197a162afac" Jul 17 18:33:00.171338 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:00.171309 2572 scope.go:117] "RemoveContainer" containerID="7979e6ae1d9fbed67bc0374ea5a711cc4bd98f1b50d0a54b9b6dc8f30fc3f985" Jul 17 18:33:00.171781 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:33:00.171548 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-w9266_odh-ai-gateway-infra(ae99619c-70a8-498a-b99f-3197a162afac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266" podUID="ae99619c-70a8-498a-b99f-3197a162afac" Jul 17 18:33:00.219942 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:00.219904 2572 scope.go:117] "RemoveContainer" containerID="131cd6f40a46b292ca5591dab1001efef2023ae9f9f06faa2c44d2d28203ba94" Jul 17 18:33:01.177109 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:01.177031 2572 generic.go:356] "Generic (PLEG): container finished" podID="9a74b088-084f-4566-b7ac-ad7409b10fc4" containerID="3718528a13f55ca1e29328f4900439bb3eeaa8ac50c5f8a7eda365f67bf7ace8" exitCode=6 Jul 17 18:33:01.177489 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:01.177102 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98" event={"ID":"9a74b088-084f-4566-b7ac-ad7409b10fc4","Type":"ContainerDied","Data":"3718528a13f55ca1e29328f4900439bb3eeaa8ac50c5f8a7eda365f67bf7ace8"} Jul 17 18:33:01.177489 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:01.177165 2572 scope.go:117] "RemoveContainer" containerID="131cd6f40a46b292ca5591dab1001efef2023ae9f9f06faa2c44d2d28203ba94" Jul 17 18:33:01.177567 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:01.177504 2572 scope.go:117] "RemoveContainer" containerID="3718528a13f55ca1e29328f4900439bb3eeaa8ac50c5f8a7eda365f67bf7ace8" Jul 17 18:33:01.177733 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:33:01.177713 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98_odh-ai-gateway-infra(9a74b088-084f-4566-b7ac-ad7409b10fc4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98" podUID="9a74b088-084f-4566-b7ac-ad7409b10fc4" Jul 17 18:33:02.201406 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:02.201373 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98"] Jul 17 18:33:02.329083 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:02.329061 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98" Jul 17 18:33:02.470547 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:02.470475 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a74b088-084f-4566-b7ac-ad7409b10fc4-maas-api-service-ca\") pod \"9a74b088-084f-4566-b7ac-ad7409b10fc4\" (UID: \"9a74b088-084f-4566-b7ac-ad7409b10fc4\") " Jul 17 18:33:02.470852 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:02.470832 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a74b088-084f-4566-b7ac-ad7409b10fc4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a74b088-084f-4566-b7ac-ad7409b10fc4" (UID: "9a74b088-084f-4566-b7ac-ad7409b10fc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:33:02.571569 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:02.571542 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a74b088-084f-4566-b7ac-ad7409b10fc4-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:33:03.187720 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:03.187686 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98" event={"ID":"9a74b088-084f-4566-b7ac-ad7409b10fc4","Type":"ContainerDied","Data":"d7bb9494ec0bc3b6f0cf6f5caabcf0d0294e4d0e6aad219c88a25624d902de06"} Jul 17 18:33:03.187720 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:03.187711 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98" Jul 17 18:33:03.187962 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:03.187744 2572 scope.go:117] "RemoveContainer" containerID="3718528a13f55ca1e29328f4900439bb3eeaa8ac50c5f8a7eda365f67bf7ace8" Jul 17 18:33:03.207838 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:03.207816 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98"] Jul 17 18:33:03.213050 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:03.213030 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hlf98"] Jul 17 18:33:03.224484 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:03.224463 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a74b088-084f-4566-b7ac-ad7409b10fc4" path="/var/lib/kubelet/pods/9a74b088-084f-4566-b7ac-ad7409b10fc4/volumes" Jul 17 18:33:10.477634 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:10.477605 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42"] Jul 17 18:33:10.478048 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:10.477901 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a74b088-084f-4566-b7ac-ad7409b10fc4" containerName="revoke-keys" Jul 17 18:33:10.478048 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:10.477912 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a74b088-084f-4566-b7ac-ad7409b10fc4" containerName="revoke-keys" Jul 17 18:33:10.478048 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:10.477928 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a74b088-084f-4566-b7ac-ad7409b10fc4" containerName="revoke-keys" Jul 17 18:33:10.478048 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:10.477934 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a74b088-084f-4566-b7ac-ad7409b10fc4" containerName="revoke-keys" Jul 17 18:33:10.478048 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:10.477941 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a74b088-084f-4566-b7ac-ad7409b10fc4" containerName="revoke-keys" Jul 17 18:33:10.478048 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:10.477946 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a74b088-084f-4566-b7ac-ad7409b10fc4" containerName="revoke-keys" Jul 17 18:33:10.478048 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:10.477991 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a74b088-084f-4566-b7ac-ad7409b10fc4" containerName="revoke-keys" Jul 17 18:33:10.478048 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:10.477999 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a74b088-084f-4566-b7ac-ad7409b10fc4" containerName="revoke-keys" Jul 17 18:33:10.482298 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:10.482282 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42" Jul 17 18:33:10.487933 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:10.487905 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42"] Jul 17 18:33:10.640708 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:10.640678 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/072099e2-22f1-4f09-a78d-22d50c6a9fb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42\" (UID: \"072099e2-22f1-4f09-a78d-22d50c6a9fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42" Jul 17 18:33:10.741524 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:10.741423 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/072099e2-22f1-4f09-a78d-22d50c6a9fb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42\" (UID: \"072099e2-22f1-4f09-a78d-22d50c6a9fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42" Jul 17 18:33:10.742071 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:10.742055 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/072099e2-22f1-4f09-a78d-22d50c6a9fb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42\" (UID: \"072099e2-22f1-4f09-a78d-22d50c6a9fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42" Jul 17 18:33:10.794276 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:10.794245 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42" Jul 17 18:33:10.915549 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:10.915524 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42"] Jul 17 18:33:10.917938 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:33:10.917911 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072099e2_22f1_4f09_a78d_22d50c6a9fb0.slice/crio-1fe4191b273fba974797ae3a2073d3cf93eded72992239f10b6ba766bf2c43e1 WatchSource:0}: Error finding container 1fe4191b273fba974797ae3a2073d3cf93eded72992239f10b6ba766bf2c43e1: Status 404 returned error can't find the container with id 1fe4191b273fba974797ae3a2073d3cf93eded72992239f10b6ba766bf2c43e1 Jul 17 18:33:11.219670 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:11.219645 2572 scope.go:117] "RemoveContainer" containerID="7979e6ae1d9fbed67bc0374ea5a711cc4bd98f1b50d0a54b9b6dc8f30fc3f985" Jul 17 18:33:11.224981 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:11.224957 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42" event={"ID":"072099e2-22f1-4f09-a78d-22d50c6a9fb0","Type":"ContainerStarted","Data":"1fe4191b273fba974797ae3a2073d3cf93eded72992239f10b6ba766bf2c43e1"} Jul 17 18:33:12.230170 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:12.230076 2572 generic.go:356] "Generic (PLEG): container finished" podID="ae99619c-70a8-498a-b99f-3197a162afac" containerID="6fca619cb3ef2e6e24a9156237f5c3ba0f76b0aad08984f61f9878c18033160c" exitCode=6 Jul 17 18:33:12.230606 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:12.230156 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266" event={"ID":"ae99619c-70a8-498a-b99f-3197a162afac","Type":"ContainerDied","Data":"6fca619cb3ef2e6e24a9156237f5c3ba0f76b0aad08984f61f9878c18033160c"} Jul 17 18:33:12.230606 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:12.230256 2572 scope.go:117] "RemoveContainer" containerID="7979e6ae1d9fbed67bc0374ea5a711cc4bd98f1b50d0a54b9b6dc8f30fc3f985" Jul 17 18:33:12.230606 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:12.230530 2572 scope.go:117] "RemoveContainer" containerID="6fca619cb3ef2e6e24a9156237f5c3ba0f76b0aad08984f61f9878c18033160c" Jul 17 18:33:12.230770 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:33:12.230747 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-w9266_odh-ai-gateway-infra(ae99619c-70a8-498a-b99f-3197a162afac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266" podUID="ae99619c-70a8-498a-b99f-3197a162afac" Jul 17 18:33:12.231736 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:12.231714 2572 generic.go:356] "Generic (PLEG): container finished" podID="072099e2-22f1-4f09-a78d-22d50c6a9fb0" containerID="d9cf89b62d59d14ec87f46d91ad4309496a7bc159149d47a930138e6a8421964" exitCode=7 Jul 17 18:33:12.231844 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:12.231782 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42" event={"ID":"072099e2-22f1-4f09-a78d-22d50c6a9fb0","Type":"ContainerDied","Data":"d9cf89b62d59d14ec87f46d91ad4309496a7bc159149d47a930138e6a8421964"} Jul 17 18:33:12.231950 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:12.231938 2572 scope.go:117] "RemoveContainer" containerID="d9cf89b62d59d14ec87f46d91ad4309496a7bc159149d47a930138e6a8421964" Jul 17 18:33:13.237231 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:13.237200 2572 generic.go:356] "Generic (PLEG): container finished" podID="072099e2-22f1-4f09-a78d-22d50c6a9fb0" containerID="45fd816576f0a6d8c472e528ee59587c5616f25f31e627b48f6c891e24b92460" exitCode=7 Jul 17 18:33:13.237679 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:13.237282 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42" event={"ID":"072099e2-22f1-4f09-a78d-22d50c6a9fb0","Type":"ContainerDied","Data":"45fd816576f0a6d8c472e528ee59587c5616f25f31e627b48f6c891e24b92460"} Jul 17 18:33:13.237679 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:13.237340 2572 scope.go:117] "RemoveContainer" containerID="d9cf89b62d59d14ec87f46d91ad4309496a7bc159149d47a930138e6a8421964" Jul 17 18:33:13.237679 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:13.237518 2572 scope.go:117] "RemoveContainer" containerID="45fd816576f0a6d8c472e528ee59587c5616f25f31e627b48f6c891e24b92460" Jul 17 18:33:13.237855 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:33:13.237709 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42_odh-ai-gateway-infra(072099e2-22f1-4f09-a78d-22d50c6a9fb0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42" podUID="072099e2-22f1-4f09-a78d-22d50c6a9fb0" Jul 17 18:33:13.259234 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:13.259208 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266"] Jul 17 18:33:13.390672 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:13.390644 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266" Jul 17 18:33:13.565813 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:13.565778 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae99619c-70a8-498a-b99f-3197a162afac-maas-api-service-ca\") pod \"ae99619c-70a8-498a-b99f-3197a162afac\" (UID: \"ae99619c-70a8-498a-b99f-3197a162afac\") " Jul 17 18:33:13.566173 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:13.566148 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae99619c-70a8-498a-b99f-3197a162afac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae99619c-70a8-498a-b99f-3197a162afac" (UID: "ae99619c-70a8-498a-b99f-3197a162afac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:33:13.666789 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:13.666759 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae99619c-70a8-498a-b99f-3197a162afac-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:33:14.244546 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:14.244515 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266" Jul 17 18:33:14.244999 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:14.244541 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266" event={"ID":"ae99619c-70a8-498a-b99f-3197a162afac","Type":"ContainerDied","Data":"888d1f3845afe44b295aea61e0a99611eafb46972c6c0c52419f259f0fef6e99"} Jul 17 18:33:14.244999 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:14.244608 2572 scope.go:117] "RemoveContainer" containerID="6fca619cb3ef2e6e24a9156237f5c3ba0f76b0aad08984f61f9878c18033160c" Jul 17 18:33:14.246405 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:14.246388 2572 scope.go:117] "RemoveContainer" containerID="45fd816576f0a6d8c472e528ee59587c5616f25f31e627b48f6c891e24b92460" Jul 17 18:33:14.246630 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:33:14.246607 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42_odh-ai-gateway-infra(072099e2-22f1-4f09-a78d-22d50c6a9fb0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42" podUID="072099e2-22f1-4f09-a78d-22d50c6a9fb0" Jul 17 18:33:14.276783 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:14.276759 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266"] Jul 17 18:33:14.280204 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:14.280184 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-w9266"] Jul 17 18:33:15.224662 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:15.224631 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae99619c-70a8-498a-b99f-3197a162afac" path="/var/lib/kubelet/pods/ae99619c-70a8-498a-b99f-3197a162afac/volumes" Jul 17 18:33:28.219768 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:28.219734 2572 scope.go:117] "RemoveContainer" containerID="45fd816576f0a6d8c472e528ee59587c5616f25f31e627b48f6c891e24b92460" Jul 17 18:33:29.310520 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:29.310486 2572 generic.go:356] "Generic (PLEG): container finished" podID="072099e2-22f1-4f09-a78d-22d50c6a9fb0" containerID="01efb577994f655b20b1cad1ee4975ca5cc89abbb9c0d51e3a13ffcac87e681e" exitCode=7 Jul 17 18:33:29.310923 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:29.310540 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42" event={"ID":"072099e2-22f1-4f09-a78d-22d50c6a9fb0","Type":"ContainerDied","Data":"01efb577994f655b20b1cad1ee4975ca5cc89abbb9c0d51e3a13ffcac87e681e"} Jul 17 18:33:29.310923 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:29.310588 2572 scope.go:117] "RemoveContainer" containerID="45fd816576f0a6d8c472e528ee59587c5616f25f31e627b48f6c891e24b92460" Jul 17 18:33:29.310923 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:29.310860 2572 scope.go:117] "RemoveContainer" containerID="01efb577994f655b20b1cad1ee4975ca5cc89abbb9c0d51e3a13ffcac87e681e" Jul 17 18:33:29.311097 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:33:29.311078 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42_odh-ai-gateway-infra(072099e2-22f1-4f09-a78d-22d50c6a9fb0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42" podUID="072099e2-22f1-4f09-a78d-22d50c6a9fb0" Jul 17 18:33:30.334311 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:30.334281 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42"] Jul 17 18:33:30.462897 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:30.462871 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42" Jul 17 18:33:30.507215 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:30.507184 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/072099e2-22f1-4f09-a78d-22d50c6a9fb0-maas-api-service-ca\") pod \"072099e2-22f1-4f09-a78d-22d50c6a9fb0\" (UID: \"072099e2-22f1-4f09-a78d-22d50c6a9fb0\") " Jul 17 18:33:30.507556 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:30.507532 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072099e2-22f1-4f09-a78d-22d50c6a9fb0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "072099e2-22f1-4f09-a78d-22d50c6a9fb0" (UID: "072099e2-22f1-4f09-a78d-22d50c6a9fb0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:33:30.607921 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:30.607851 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/072099e2-22f1-4f09-a78d-22d50c6a9fb0-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:33:31.321655 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:31.321623 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42" event={"ID":"072099e2-22f1-4f09-a78d-22d50c6a9fb0","Type":"ContainerDied","Data":"1fe4191b273fba974797ae3a2073d3cf93eded72992239f10b6ba766bf2c43e1"} Jul 17 18:33:31.321655 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:31.321646 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42" Jul 17 18:33:31.321892 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:31.321680 2572 scope.go:117] "RemoveContainer" containerID="01efb577994f655b20b1cad1ee4975ca5cc89abbb9c0d51e3a13ffcac87e681e" Jul 17 18:33:31.338500 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:31.338473 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42"] Jul 17 18:33:31.344287 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:31.344263 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8qp42"] Jul 17 18:33:33.225491 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:33.225458 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="072099e2-22f1-4f09-a78d-22d50c6a9fb0" path="/var/lib/kubelet/pods/072099e2-22f1-4f09-a78d-22d50c6a9fb0/volumes" Jul 17 18:33:47.001646 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.001610 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn"] Jul 17 18:33:47.002040 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.001943 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae99619c-70a8-498a-b99f-3197a162afac" containerName="revoke-keys" Jul 17 18:33:47.002040 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.001956 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae99619c-70a8-498a-b99f-3197a162afac" containerName="revoke-keys" Jul 17 18:33:47.002040 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.001975 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="072099e2-22f1-4f09-a78d-22d50c6a9fb0" containerName="revoke-keys" Jul 17 18:33:47.002040 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.001981 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="072099e2-22f1-4f09-a78d-22d50c6a9fb0" containerName="revoke-keys" Jul 17 18:33:47.002040 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.001992 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae99619c-70a8-498a-b99f-3197a162afac" containerName="revoke-keys" Jul 17 18:33:47.002040 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.002000 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae99619c-70a8-498a-b99f-3197a162afac" containerName="revoke-keys" Jul 17 18:33:47.002040 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.002024 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="072099e2-22f1-4f09-a78d-22d50c6a9fb0" containerName="revoke-keys" Jul 17 18:33:47.002040 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.002030 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="072099e2-22f1-4f09-a78d-22d50c6a9fb0" containerName="revoke-keys" Jul 17 18:33:47.002040 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.002036 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="072099e2-22f1-4f09-a78d-22d50c6a9fb0" containerName="revoke-keys" Jul 17 18:33:47.002040 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.002041 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="072099e2-22f1-4f09-a78d-22d50c6a9fb0" containerName="revoke-keys" Jul 17 18:33:47.002350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.002089 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae99619c-70a8-498a-b99f-3197a162afac" containerName="revoke-keys" Jul 17 18:33:47.002350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.002098 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="072099e2-22f1-4f09-a78d-22d50c6a9fb0" containerName="revoke-keys" Jul 17 18:33:47.002350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.002104 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae99619c-70a8-498a-b99f-3197a162afac" containerName="revoke-keys" Jul 17 18:33:47.002350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.002110 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="072099e2-22f1-4f09-a78d-22d50c6a9fb0" containerName="revoke-keys" Jul 17 18:33:47.002350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.002116 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="072099e2-22f1-4f09-a78d-22d50c6a9fb0" containerName="revoke-keys" Jul 17 18:33:47.002350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.002123 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a74b088-084f-4566-b7ac-ad7409b10fc4" containerName="revoke-keys" Jul 17 18:33:47.006430 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.006414 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn" Jul 17 18:33:47.008762 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.008740 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:33:47.008857 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.008797 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:33:47.011885 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.011859 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn"] Jul 17 18:33:47.048744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.048720 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/699f9604-52ab-4bac-8e03-98dfad6e542b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn\" (UID: \"699f9604-52ab-4bac-8e03-98dfad6e542b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn" Jul 17 18:33:47.149389 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.149357 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/699f9604-52ab-4bac-8e03-98dfad6e542b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn\" (UID: \"699f9604-52ab-4bac-8e03-98dfad6e542b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn" Jul 17 18:33:47.149989 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.149968 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/699f9604-52ab-4bac-8e03-98dfad6e542b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn\" (UID: \"699f9604-52ab-4bac-8e03-98dfad6e542b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn" Jul 17 18:33:47.317797 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.317766 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn" Jul 17 18:33:47.444634 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:47.444610 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn"] Jul 17 18:33:47.446481 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:33:47.446452 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699f9604_52ab_4bac_8e03_98dfad6e542b.slice/crio-716476d05a34b18f9deee23e5a727903ad83c48b27ad5da5e4478ddaf1fc9b94 WatchSource:0}: Error finding container 716476d05a34b18f9deee23e5a727903ad83c48b27ad5da5e4478ddaf1fc9b94: Status 404 returned error can't find the container with id 716476d05a34b18f9deee23e5a727903ad83c48b27ad5da5e4478ddaf1fc9b94 Jul 17 18:33:48.395224 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:48.395193 2572 generic.go:356] "Generic (PLEG): container finished" podID="699f9604-52ab-4bac-8e03-98dfad6e542b" containerID="c347d7ebe0c712a005d59b32aefc6d47b9740788b74ef8195f433b58b27f1879" exitCode=6 Jul 17 18:33:48.395584 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:48.395241 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn" event={"ID":"699f9604-52ab-4bac-8e03-98dfad6e542b","Type":"ContainerDied","Data":"c347d7ebe0c712a005d59b32aefc6d47b9740788b74ef8195f433b58b27f1879"} Jul 17 18:33:48.395584 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:48.395281 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn" event={"ID":"699f9604-52ab-4bac-8e03-98dfad6e542b","Type":"ContainerStarted","Data":"716476d05a34b18f9deee23e5a727903ad83c48b27ad5da5e4478ddaf1fc9b94"} Jul 17 18:33:48.395584 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:48.395441 2572 scope.go:117] "RemoveContainer" containerID="c347d7ebe0c712a005d59b32aefc6d47b9740788b74ef8195f433b58b27f1879" Jul 17 18:33:49.401076 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:49.400982 2572 generic.go:356] "Generic (PLEG): container finished" podID="699f9604-52ab-4bac-8e03-98dfad6e542b" containerID="4b373467e5cadbc45cc517718b2960db4264272f8cdddcb756bfa331d43657a3" exitCode=6 Jul 17 18:33:49.401076 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:49.401041 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn" event={"ID":"699f9604-52ab-4bac-8e03-98dfad6e542b","Type":"ContainerDied","Data":"4b373467e5cadbc45cc517718b2960db4264272f8cdddcb756bfa331d43657a3"} Jul 17 18:33:49.401518 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:49.401102 2572 scope.go:117] "RemoveContainer" containerID="c347d7ebe0c712a005d59b32aefc6d47b9740788b74ef8195f433b58b27f1879" Jul 17 18:33:49.401518 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:49.401218 2572 scope.go:117] "RemoveContainer" containerID="4b373467e5cadbc45cc517718b2960db4264272f8cdddcb756bfa331d43657a3" Jul 17 18:33:49.401518 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:33:49.401462 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn_odh-ai-gateway-infra(699f9604-52ab-4bac-8e03-98dfad6e542b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn" podUID="699f9604-52ab-4bac-8e03-98dfad6e542b" Jul 17 18:33:50.406417 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:33:50.406394 2572 scope.go:117] "RemoveContainer" containerID="4b373467e5cadbc45cc517718b2960db4264272f8cdddcb756bfa331d43657a3" Jul 17 18:33:50.406800 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:33:50.406576 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn_odh-ai-gateway-infra(699f9604-52ab-4bac-8e03-98dfad6e542b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn" podUID="699f9604-52ab-4bac-8e03-98dfad6e542b" Jul 17 18:34:04.220190 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:04.220162 2572 scope.go:117] "RemoveContainer" containerID="4b373467e5cadbc45cc517718b2960db4264272f8cdddcb756bfa331d43657a3" Jul 17 18:34:04.461919 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:04.461895 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn" event={"ID":"699f9604-52ab-4bac-8e03-98dfad6e542b","Type":"ContainerStarted","Data":"9e5be3b25005077e6724acf27101dc52540c4a7b5c5c6d920ab16fbd6e863a3f"} Jul 17 18:34:04.481242 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:04.481164 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn" podStartSLOduration=18.481150286 podStartE2EDuration="18.481150286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:34:04.479745268 +0000 UTC m=+2239.889172874" watchObservedRunningTime="2026-07-17 18:34:04.481150286 +0000 UTC m=+2239.890577893" Jul 17 18:34:05.467623 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:05.467592 2572 generic.go:356] "Generic (PLEG): container finished" podID="699f9604-52ab-4bac-8e03-98dfad6e542b" containerID="9e5be3b25005077e6724acf27101dc52540c4a7b5c5c6d920ab16fbd6e863a3f" exitCode=6 Jul 17 18:34:05.468054 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:05.467647 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn" event={"ID":"699f9604-52ab-4bac-8e03-98dfad6e542b","Type":"ContainerDied","Data":"9e5be3b25005077e6724acf27101dc52540c4a7b5c5c6d920ab16fbd6e863a3f"} Jul 17 18:34:05.468054 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:05.467693 2572 scope.go:117] "RemoveContainer" containerID="4b373467e5cadbc45cc517718b2960db4264272f8cdddcb756bfa331d43657a3" Jul 17 18:34:05.468054 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:05.468039 2572 scope.go:117] "RemoveContainer" containerID="9e5be3b25005077e6724acf27101dc52540c4a7b5c5c6d920ab16fbd6e863a3f" Jul 17 18:34:05.468276 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:34:05.468253 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn_odh-ai-gateway-infra(699f9604-52ab-4bac-8e03-98dfad6e542b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn" podUID="699f9604-52ab-4bac-8e03-98dfad6e542b" Jul 17 18:34:05.492769 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:05.492740 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn"] Jul 17 18:34:06.601485 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:06.601455 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn" Jul 17 18:34:06.718958 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:06.718931 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/699f9604-52ab-4bac-8e03-98dfad6e542b-maas-api-service-ca\") pod \"699f9604-52ab-4bac-8e03-98dfad6e542b\" (UID: \"699f9604-52ab-4bac-8e03-98dfad6e542b\") " Jul 17 18:34:06.719325 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:06.719301 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699f9604-52ab-4bac-8e03-98dfad6e542b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "699f9604-52ab-4bac-8e03-98dfad6e542b" (UID: "699f9604-52ab-4bac-8e03-98dfad6e542b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:34:06.820065 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:06.820035 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/699f9604-52ab-4bac-8e03-98dfad6e542b-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:34:07.477640 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:07.477608 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn" Jul 17 18:34:07.477640 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:07.477613 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn" event={"ID":"699f9604-52ab-4bac-8e03-98dfad6e542b","Type":"ContainerDied","Data":"716476d05a34b18f9deee23e5a727903ad83c48b27ad5da5e4478ddaf1fc9b94"} Jul 17 18:34:07.477900 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:07.477687 2572 scope.go:117] "RemoveContainer" containerID="9e5be3b25005077e6724acf27101dc52540c4a7b5c5c6d920ab16fbd6e863a3f" Jul 17 18:34:07.493716 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:07.493662 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn"] Jul 17 18:34:07.500844 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:07.500808 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-qq6cn"] Jul 17 18:34:09.224777 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:09.224747 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="699f9604-52ab-4bac-8e03-98dfad6e542b" path="/var/lib/kubelet/pods/699f9604-52ab-4bac-8e03-98dfad6e542b/volumes" Jul 17 18:34:10.528537 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.528508 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw"] Jul 17 18:34:10.528893 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.528810 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="699f9604-52ab-4bac-8e03-98dfad6e542b" containerName="revoke-keys" Jul 17 18:34:10.528893 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.528822 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="699f9604-52ab-4bac-8e03-98dfad6e542b" containerName="revoke-keys" Jul 17 18:34:10.528893 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.528842 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae99619c-70a8-498a-b99f-3197a162afac" containerName="revoke-keys" Jul 17 18:34:10.528893 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.528851 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae99619c-70a8-498a-b99f-3197a162afac" containerName="revoke-keys" Jul 17 18:34:10.528893 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.528862 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="699f9604-52ab-4bac-8e03-98dfad6e542b" containerName="revoke-keys" Jul 17 18:34:10.528893 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.528867 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="699f9604-52ab-4bac-8e03-98dfad6e542b" containerName="revoke-keys" Jul 17 18:34:10.529104 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.528913 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="699f9604-52ab-4bac-8e03-98dfad6e542b" containerName="revoke-keys" Jul 17 18:34:10.529104 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.528922 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="699f9604-52ab-4bac-8e03-98dfad6e542b" containerName="revoke-keys" Jul 17 18:34:10.529104 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.528931 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae99619c-70a8-498a-b99f-3197a162afac" containerName="revoke-keys" Jul 17 18:34:10.533314 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.533298 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw" Jul 17 18:34:10.535810 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.535788 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:34:10.536097 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.536076 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:34:10.539247 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.539207 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw"] Jul 17 18:34:10.654848 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.654818 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/333ac567-b7f0-4f1c-bec2-1c2b48345b35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw\" (UID: \"333ac567-b7f0-4f1c-bec2-1c2b48345b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw" Jul 17 18:34:10.755450 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.755423 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/333ac567-b7f0-4f1c-bec2-1c2b48345b35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw\" (UID: \"333ac567-b7f0-4f1c-bec2-1c2b48345b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw" Jul 17 18:34:10.756026 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.755990 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/333ac567-b7f0-4f1c-bec2-1c2b48345b35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw\" (UID: \"333ac567-b7f0-4f1c-bec2-1c2b48345b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw" Jul 17 18:34:10.844499 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.844438 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw" Jul 17 18:34:10.969912 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:10.969889 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw"] Jul 17 18:34:10.972210 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:34:10.972179 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333ac567_b7f0_4f1c_bec2_1c2b48345b35.slice/crio-c976c2abdb2b7a7ad80fc2642ccb26fa9edbfab8aa38802bc0f5ae0c2e01e701 WatchSource:0}: Error finding container c976c2abdb2b7a7ad80fc2642ccb26fa9edbfab8aa38802bc0f5ae0c2e01e701: Status 404 returned error can't find the container with id c976c2abdb2b7a7ad80fc2642ccb26fa9edbfab8aa38802bc0f5ae0c2e01e701 Jul 17 18:34:11.496278 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:11.496194 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw" event={"ID":"333ac567-b7f0-4f1c-bec2-1c2b48345b35","Type":"ContainerStarted","Data":"389b8048a855b9a29884bdad3a3496396554e6918a1601053697a231bfaf051b"} Jul 17 18:34:11.496278 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:11.496244 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw" event={"ID":"333ac567-b7f0-4f1c-bec2-1c2b48345b35","Type":"ContainerStarted","Data":"c976c2abdb2b7a7ad80fc2642ccb26fa9edbfab8aa38802bc0f5ae0c2e01e701"} Jul 17 18:34:11.515719 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:11.515676 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw" podStartSLOduration=1.515660993 podStartE2EDuration="1.515660993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:34:11.513568289 +0000 UTC m=+2246.922995895" watchObservedRunningTime="2026-07-17 18:34:11.515660993 +0000 UTC m=+2246.925088654" Jul 17 18:34:12.502084 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:12.502048 2572 generic.go:356] "Generic (PLEG): container finished" podID="333ac567-b7f0-4f1c-bec2-1c2b48345b35" containerID="389b8048a855b9a29884bdad3a3496396554e6918a1601053697a231bfaf051b" exitCode=7 Jul 17 18:34:12.502445 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:12.502091 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw" event={"ID":"333ac567-b7f0-4f1c-bec2-1c2b48345b35","Type":"ContainerDied","Data":"389b8048a855b9a29884bdad3a3496396554e6918a1601053697a231bfaf051b"} Jul 17 18:34:12.502445 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:12.502289 2572 scope.go:117] "RemoveContainer" containerID="389b8048a855b9a29884bdad3a3496396554e6918a1601053697a231bfaf051b" Jul 17 18:34:13.507570 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:13.507490 2572 generic.go:356] "Generic (PLEG): container finished" podID="333ac567-b7f0-4f1c-bec2-1c2b48345b35" containerID="3ba811a2c02e8f2d58ea6b18836cf2446a54b0bdacd52feb44d3ddffdccbe8ba" exitCode=7 Jul 17 18:34:13.507570 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:13.507534 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw" event={"ID":"333ac567-b7f0-4f1c-bec2-1c2b48345b35","Type":"ContainerDied","Data":"3ba811a2c02e8f2d58ea6b18836cf2446a54b0bdacd52feb44d3ddffdccbe8ba"} Jul 17 18:34:13.507985 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:13.507576 2572 scope.go:117] "RemoveContainer" containerID="389b8048a855b9a29884bdad3a3496396554e6918a1601053697a231bfaf051b" Jul 17 18:34:13.507985 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:13.507753 2572 scope.go:117] "RemoveContainer" containerID="3ba811a2c02e8f2d58ea6b18836cf2446a54b0bdacd52feb44d3ddffdccbe8ba" Jul 17 18:34:13.508090 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:34:13.508001 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw_odh-ai-gateway-infra(333ac567-b7f0-4f1c-bec2-1c2b48345b35)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw" podUID="333ac567-b7f0-4f1c-bec2-1c2b48345b35" Jul 17 18:34:14.513285 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:14.513250 2572 scope.go:117] "RemoveContainer" containerID="3ba811a2c02e8f2d58ea6b18836cf2446a54b0bdacd52feb44d3ddffdccbe8ba" Jul 17 18:34:14.513670 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:34:14.513415 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw_odh-ai-gateway-infra(333ac567-b7f0-4f1c-bec2-1c2b48345b35)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw" podUID="333ac567-b7f0-4f1c-bec2-1c2b48345b35" Jul 17 18:34:27.224792 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:27.224763 2572 scope.go:117] "RemoveContainer" containerID="3ba811a2c02e8f2d58ea6b18836cf2446a54b0bdacd52feb44d3ddffdccbe8ba" Jul 17 18:34:27.566146 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:27.566118 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw" event={"ID":"333ac567-b7f0-4f1c-bec2-1c2b48345b35","Type":"ContainerStarted","Data":"46fafbcd92b7da5429829ca1197f34495ac3d9bb1d774577cefb374584c72853"} Jul 17 18:34:28.571926 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:28.571895 2572 generic.go:356] "Generic (PLEG): container finished" podID="333ac567-b7f0-4f1c-bec2-1c2b48345b35" containerID="46fafbcd92b7da5429829ca1197f34495ac3d9bb1d774577cefb374584c72853" exitCode=7 Jul 17 18:34:28.572378 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:28.571971 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw" event={"ID":"333ac567-b7f0-4f1c-bec2-1c2b48345b35","Type":"ContainerDied","Data":"46fafbcd92b7da5429829ca1197f34495ac3d9bb1d774577cefb374584c72853"} Jul 17 18:34:28.572378 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:28.572043 2572 scope.go:117] "RemoveContainer" containerID="3ba811a2c02e8f2d58ea6b18836cf2446a54b0bdacd52feb44d3ddffdccbe8ba" Jul 17 18:34:28.572378 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:28.572286 2572 scope.go:117] "RemoveContainer" containerID="46fafbcd92b7da5429829ca1197f34495ac3d9bb1d774577cefb374584c72853" Jul 17 18:34:28.572549 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:34:28.572512 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw_odh-ai-gateway-infra(333ac567-b7f0-4f1c-bec2-1c2b48345b35)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw" podUID="333ac567-b7f0-4f1c-bec2-1c2b48345b35" Jul 17 18:34:28.593762 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:28.593737 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw"] Jul 17 18:34:29.702071 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:29.702050 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw" Jul 17 18:34:29.810030 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:29.809977 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/333ac567-b7f0-4f1c-bec2-1c2b48345b35-maas-api-service-ca\") pod \"333ac567-b7f0-4f1c-bec2-1c2b48345b35\" (UID: \"333ac567-b7f0-4f1c-bec2-1c2b48345b35\") " Jul 17 18:34:29.810320 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:29.810297 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333ac567-b7f0-4f1c-bec2-1c2b48345b35-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "333ac567-b7f0-4f1c-bec2-1c2b48345b35" (UID: "333ac567-b7f0-4f1c-bec2-1c2b48345b35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:34:29.911071 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:29.910980 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/333ac567-b7f0-4f1c-bec2-1c2b48345b35-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:34:30.583120 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:30.583087 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw" event={"ID":"333ac567-b7f0-4f1c-bec2-1c2b48345b35","Type":"ContainerDied","Data":"c976c2abdb2b7a7ad80fc2642ccb26fa9edbfab8aa38802bc0f5ae0c2e01e701"} Jul 17 18:34:30.583307 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:30.583136 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw" Jul 17 18:34:30.583307 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:30.583146 2572 scope.go:117] "RemoveContainer" containerID="46fafbcd92b7da5429829ca1197f34495ac3d9bb1d774577cefb374584c72853" Jul 17 18:34:30.604482 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:30.604454 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw"] Jul 17 18:34:30.609269 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:30.609249 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-d7xrw"] Jul 17 18:34:31.224413 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:31.224383 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="333ac567-b7f0-4f1c-bec2-1c2b48345b35" path="/var/lib/kubelet/pods/333ac567-b7f0-4f1c-bec2-1c2b48345b35/volumes" Jul 17 18:34:47.055317 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.055283 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw"] Jul 17 18:34:47.055708 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.055558 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="333ac567-b7f0-4f1c-bec2-1c2b48345b35" containerName="revoke-keys" Jul 17 18:34:47.055708 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.055568 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="333ac567-b7f0-4f1c-bec2-1c2b48345b35" containerName="revoke-keys" Jul 17 18:34:47.055708 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.055577 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="333ac567-b7f0-4f1c-bec2-1c2b48345b35" containerName="revoke-keys" Jul 17 18:34:47.055708 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.055584 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="333ac567-b7f0-4f1c-bec2-1c2b48345b35" containerName="revoke-keys" Jul 17 18:34:47.055708 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.055593 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="699f9604-52ab-4bac-8e03-98dfad6e542b" containerName="revoke-keys" Jul 17 18:34:47.055708 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.055598 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="699f9604-52ab-4bac-8e03-98dfad6e542b" containerName="revoke-keys" Jul 17 18:34:47.055708 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.055605 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="333ac567-b7f0-4f1c-bec2-1c2b48345b35" containerName="revoke-keys" Jul 17 18:34:47.055708 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.055610 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="333ac567-b7f0-4f1c-bec2-1c2b48345b35" containerName="revoke-keys" Jul 17 18:34:47.055708 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.055658 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="333ac567-b7f0-4f1c-bec2-1c2b48345b35" containerName="revoke-keys" Jul 17 18:34:47.055708 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.055665 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="699f9604-52ab-4bac-8e03-98dfad6e542b" containerName="revoke-keys" Jul 17 18:34:47.055708 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.055672 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="333ac567-b7f0-4f1c-bec2-1c2b48345b35" containerName="revoke-keys" Jul 17 18:34:47.060166 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.060149 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw" Jul 17 18:34:47.062888 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.062867 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:34:47.063195 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.063060 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:34:47.066745 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.066723 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw"] Jul 17 18:34:47.158046 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.157990 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b69d0f-3ec5-41df-b11f-191aff5cde7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw\" (UID: \"c8b69d0f-3ec5-41df-b11f-191aff5cde7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw" Jul 17 18:34:47.259465 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.259438 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b69d0f-3ec5-41df-b11f-191aff5cde7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw\" (UID: \"c8b69d0f-3ec5-41df-b11f-191aff5cde7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw" Jul 17 18:34:47.260085 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.260062 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b69d0f-3ec5-41df-b11f-191aff5cde7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw\" (UID: \"c8b69d0f-3ec5-41df-b11f-191aff5cde7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw" Jul 17 18:34:47.371397 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.371315 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw" Jul 17 18:34:47.501285 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.501262 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw"] Jul 17 18:34:47.503664 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:34:47.503632 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b69d0f_3ec5_41df_b11f_191aff5cde7c.slice/crio-7bcd18206749b767d861bf1bc91f10e58caca95d71191c4c8fe33f41d868af98 WatchSource:0}: Error finding container 7bcd18206749b767d861bf1bc91f10e58caca95d71191c4c8fe33f41d868af98: Status 404 returned error can't find the container with id 7bcd18206749b767d861bf1bc91f10e58caca95d71191c4c8fe33f41d868af98 Jul 17 18:34:47.658437 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:47.658344 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw" event={"ID":"c8b69d0f-3ec5-41df-b11f-191aff5cde7c","Type":"ContainerStarted","Data":"7bcd18206749b767d861bf1bc91f10e58caca95d71191c4c8fe33f41d868af98"} Jul 17 18:34:48.663668 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:48.663634 2572 generic.go:356] "Generic (PLEG): container finished" podID="c8b69d0f-3ec5-41df-b11f-191aff5cde7c" containerID="f22a597d43e99ee3a8733e0846b05a34e780590f10017e80a558225a3dff26bb" exitCode=6 Jul 17 18:34:48.664070 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:48.663723 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw" event={"ID":"c8b69d0f-3ec5-41df-b11f-191aff5cde7c","Type":"ContainerDied","Data":"f22a597d43e99ee3a8733e0846b05a34e780590f10017e80a558225a3dff26bb"} Jul 17 18:34:48.664070 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:48.663899 2572 scope.go:117] "RemoveContainer" containerID="f22a597d43e99ee3a8733e0846b05a34e780590f10017e80a558225a3dff26bb" Jul 17 18:34:49.670156 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:49.670121 2572 generic.go:356] "Generic (PLEG): container finished" podID="c8b69d0f-3ec5-41df-b11f-191aff5cde7c" containerID="29c1f63802f6a4139b2b14afc37a9c1eb32c8ab7b7e0717ae2423950e50db221" exitCode=6 Jul 17 18:34:49.670525 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:49.670206 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw" event={"ID":"c8b69d0f-3ec5-41df-b11f-191aff5cde7c","Type":"ContainerDied","Data":"29c1f63802f6a4139b2b14afc37a9c1eb32c8ab7b7e0717ae2423950e50db221"} Jul 17 18:34:49.670525 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:49.670263 2572 scope.go:117] "RemoveContainer" containerID="f22a597d43e99ee3a8733e0846b05a34e780590f10017e80a558225a3dff26bb" Jul 17 18:34:49.670525 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:49.670467 2572 scope.go:117] "RemoveContainer" containerID="29c1f63802f6a4139b2b14afc37a9c1eb32c8ab7b7e0717ae2423950e50db221" Jul 17 18:34:49.670681 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:34:49.670664 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw_odh-ai-gateway-infra(c8b69d0f-3ec5-41df-b11f-191aff5cde7c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw" podUID="c8b69d0f-3ec5-41df-b11f-191aff5cde7c" Jul 17 18:34:50.675894 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:34:50.675860 2572 scope.go:117] "RemoveContainer" containerID="29c1f63802f6a4139b2b14afc37a9c1eb32c8ab7b7e0717ae2423950e50db221" Jul 17 18:34:50.676494 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:34:50.676103 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw_odh-ai-gateway-infra(c8b69d0f-3ec5-41df-b11f-191aff5cde7c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw" podUID="c8b69d0f-3ec5-41df-b11f-191aff5cde7c" Jul 17 18:35:02.220302 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:02.220268 2572 scope.go:117] "RemoveContainer" containerID="29c1f63802f6a4139b2b14afc37a9c1eb32c8ab7b7e0717ae2423950e50db221" Jul 17 18:35:02.732861 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:02.732829 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw" event={"ID":"c8b69d0f-3ec5-41df-b11f-191aff5cde7c","Type":"ContainerStarted","Data":"1c664e8b5bc6c57f4c77ca74b7c9aed058eda09d0d4bca14c14f4cad2734b419"} Jul 17 18:35:02.748382 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:02.748335 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw" podStartSLOduration=15.748320481 podStartE2EDuration="15.748320481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:35:02.746428588 +0000 UTC m=+2298.155856204" watchObservedRunningTime="2026-07-17 18:35:02.748320481 +0000 UTC m=+2298.157748086" Jul 17 18:35:03.103308 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:03.103279 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq"] Jul 17 18:35:03.103630 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:03.103618 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="333ac567-b7f0-4f1c-bec2-1c2b48345b35" containerName="revoke-keys" Jul 17 18:35:03.106436 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:03.106419 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq" Jul 17 18:35:03.114060 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:03.114037 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq"] Jul 17 18:35:03.192915 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:03.192881 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2afaa3-8e69-4274-accf-c4f89b655592-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-595sq\" (UID: \"2c2afaa3-8e69-4274-accf-c4f89b655592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq" Jul 17 18:35:03.293520 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:03.293488 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2afaa3-8e69-4274-accf-c4f89b655592-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-595sq\" (UID: \"2c2afaa3-8e69-4274-accf-c4f89b655592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq" Jul 17 18:35:03.294250 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:03.294227 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2afaa3-8e69-4274-accf-c4f89b655592-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-595sq\" (UID: \"2c2afaa3-8e69-4274-accf-c4f89b655592\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq" Jul 17 18:35:03.417400 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:03.417321 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq" Jul 17 18:35:03.541053 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:03.541027 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq"] Jul 17 18:35:03.543473 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:35:03.543445 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2afaa3_8e69_4274_accf_c4f89b655592.slice/crio-16ba41e6ebf7c90907eb1458aa297fc895a2c454a8189ee0b1019a7a94b6310c WatchSource:0}: Error finding container 16ba41e6ebf7c90907eb1458aa297fc895a2c454a8189ee0b1019a7a94b6310c: Status 404 returned error can't find the container with id 16ba41e6ebf7c90907eb1458aa297fc895a2c454a8189ee0b1019a7a94b6310c Jul 17 18:35:03.739251 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:03.739172 2572 generic.go:356] "Generic (PLEG): container finished" podID="c8b69d0f-3ec5-41df-b11f-191aff5cde7c" containerID="1c664e8b5bc6c57f4c77ca74b7c9aed058eda09d0d4bca14c14f4cad2734b419" exitCode=6 Jul 17 18:35:03.739401 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:03.739246 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw" event={"ID":"c8b69d0f-3ec5-41df-b11f-191aff5cde7c","Type":"ContainerDied","Data":"1c664e8b5bc6c57f4c77ca74b7c9aed058eda09d0d4bca14c14f4cad2734b419"} Jul 17 18:35:03.739401 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:03.739308 2572 scope.go:117] "RemoveContainer" containerID="29c1f63802f6a4139b2b14afc37a9c1eb32c8ab7b7e0717ae2423950e50db221" Jul 17 18:35:03.739601 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:03.739574 2572 scope.go:117] "RemoveContainer" containerID="1c664e8b5bc6c57f4c77ca74b7c9aed058eda09d0d4bca14c14f4cad2734b419" Jul 17 18:35:03.739838 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:35:03.739812 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw_odh-ai-gateway-infra(c8b69d0f-3ec5-41df-b11f-191aff5cde7c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw" podUID="c8b69d0f-3ec5-41df-b11f-191aff5cde7c" Jul 17 18:35:03.740672 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:03.740650 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq" event={"ID":"2c2afaa3-8e69-4274-accf-c4f89b655592","Type":"ContainerStarted","Data":"16ba41e6ebf7c90907eb1458aa297fc895a2c454a8189ee0b1019a7a94b6310c"} Jul 17 18:35:03.756055 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:03.755966 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw"] Jul 17 18:35:04.748678 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:04.748650 2572 generic.go:356] "Generic (PLEG): container finished" podID="2c2afaa3-8e69-4274-accf-c4f89b655592" containerID="fcba74dfe6760fd112378c267f6505ef401f9535c3224dd503e727ec6e78996a" exitCode=6 Jul 17 18:35:04.748990 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:04.748699 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq" event={"ID":"2c2afaa3-8e69-4274-accf-c4f89b655592","Type":"ContainerDied","Data":"fcba74dfe6760fd112378c267f6505ef401f9535c3224dd503e727ec6e78996a"} Jul 17 18:35:04.748990 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:04.748924 2572 scope.go:117] "RemoveContainer" containerID="fcba74dfe6760fd112378c267f6505ef401f9535c3224dd503e727ec6e78996a" Jul 17 18:35:04.867804 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:04.867778 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw" Jul 17 18:35:05.008365 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:05.008311 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b69d0f-3ec5-41df-b11f-191aff5cde7c-maas-api-service-ca\") pod \"c8b69d0f-3ec5-41df-b11f-191aff5cde7c\" (UID: \"c8b69d0f-3ec5-41df-b11f-191aff5cde7c\") " Jul 17 18:35:05.008629 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:05.008609 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b69d0f-3ec5-41df-b11f-191aff5cde7c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c8b69d0f-3ec5-41df-b11f-191aff5cde7c" (UID: "c8b69d0f-3ec5-41df-b11f-191aff5cde7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:35:05.109454 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:05.109428 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b69d0f-3ec5-41df-b11f-191aff5cde7c-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:35:05.753377 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:05.753288 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw" Jul 17 18:35:05.753828 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:05.753280 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw" event={"ID":"c8b69d0f-3ec5-41df-b11f-191aff5cde7c","Type":"ContainerDied","Data":"7bcd18206749b767d861bf1bc91f10e58caca95d71191c4c8fe33f41d868af98"} Jul 17 18:35:05.753828 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:05.753477 2572 scope.go:117] "RemoveContainer" containerID="1c664e8b5bc6c57f4c77ca74b7c9aed058eda09d0d4bca14c14f4cad2734b419" Jul 17 18:35:05.754982 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:05.754962 2572 generic.go:356] "Generic (PLEG): container finished" podID="2c2afaa3-8e69-4274-accf-c4f89b655592" containerID="bf4dff002d3f0c9bbf983898b5eb1db4624e0344740d022e0b496bb36170e98f" exitCode=6 Jul 17 18:35:05.755094 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:05.754993 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq" event={"ID":"2c2afaa3-8e69-4274-accf-c4f89b655592","Type":"ContainerDied","Data":"bf4dff002d3f0c9bbf983898b5eb1db4624e0344740d022e0b496bb36170e98f"} Jul 17 18:35:05.755248 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:05.755234 2572 scope.go:117] "RemoveContainer" containerID="bf4dff002d3f0c9bbf983898b5eb1db4624e0344740d022e0b496bb36170e98f" Jul 17 18:35:05.755427 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:35:05.755408 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-595sq_odh-ai-gateway-infra(2c2afaa3-8e69-4274-accf-c4f89b655592)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq" podUID="2c2afaa3-8e69-4274-accf-c4f89b655592" Jul 17 18:35:05.761315 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:05.761296 2572 scope.go:117] "RemoveContainer" containerID="fcba74dfe6760fd112378c267f6505ef401f9535c3224dd503e727ec6e78996a" Jul 17 18:35:05.786633 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:05.786612 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw"] Jul 17 18:35:05.792605 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:05.792583 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lqzrw"] Jul 17 18:35:06.761262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:06.761231 2572 scope.go:117] "RemoveContainer" containerID="bf4dff002d3f0c9bbf983898b5eb1db4624e0344740d022e0b496bb36170e98f" Jul 17 18:35:06.761708 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:35:06.761480 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-595sq_odh-ai-gateway-infra(2c2afaa3-8e69-4274-accf-c4f89b655592)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq" podUID="2c2afaa3-8e69-4274-accf-c4f89b655592" Jul 17 18:35:07.224266 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:07.224234 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8b69d0f-3ec5-41df-b11f-191aff5cde7c" path="/var/lib/kubelet/pods/c8b69d0f-3ec5-41df-b11f-191aff5cde7c/volumes" Jul 17 18:35:18.220146 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:18.220114 2572 scope.go:117] "RemoveContainer" containerID="bf4dff002d3f0c9bbf983898b5eb1db4624e0344740d022e0b496bb36170e98f" Jul 17 18:35:18.813577 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:18.813541 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq" event={"ID":"2c2afaa3-8e69-4274-accf-c4f89b655592","Type":"ContainerStarted","Data":"87b0578e64cf6f63d282bcaa6120e1558982da8556974548304fdd29f19672c7"} Jul 17 18:35:18.830617 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:18.830570 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq" podStartSLOduration=15.830557671 podStartE2EDuration="15.830557671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:35:18.828575935 +0000 UTC m=+2314.238003532" watchObservedRunningTime="2026-07-17 18:35:18.830557671 +0000 UTC m=+2314.239985298" Jul 17 18:35:19.819314 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:19.819281 2572 generic.go:356] "Generic (PLEG): container finished" podID="2c2afaa3-8e69-4274-accf-c4f89b655592" containerID="87b0578e64cf6f63d282bcaa6120e1558982da8556974548304fdd29f19672c7" exitCode=6 Jul 17 18:35:19.819748 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:19.819356 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq" event={"ID":"2c2afaa3-8e69-4274-accf-c4f89b655592","Type":"ContainerDied","Data":"87b0578e64cf6f63d282bcaa6120e1558982da8556974548304fdd29f19672c7"} Jul 17 18:35:19.819748 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:19.819418 2572 scope.go:117] "RemoveContainer" containerID="bf4dff002d3f0c9bbf983898b5eb1db4624e0344740d022e0b496bb36170e98f" Jul 17 18:35:19.819748 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:19.819669 2572 scope.go:117] "RemoveContainer" containerID="87b0578e64cf6f63d282bcaa6120e1558982da8556974548304fdd29f19672c7" Jul 17 18:35:19.819895 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:35:19.819877 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-595sq_odh-ai-gateway-infra(2c2afaa3-8e69-4274-accf-c4f89b655592)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq" podUID="2c2afaa3-8e69-4274-accf-c4f89b655592" Jul 17 18:35:19.838540 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:19.838515 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq"] Jul 17 18:35:20.950634 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:20.950610 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq" Jul 17 18:35:21.040332 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:21.040291 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2afaa3-8e69-4274-accf-c4f89b655592-maas-api-service-ca\") pod \"2c2afaa3-8e69-4274-accf-c4f89b655592\" (UID: \"2c2afaa3-8e69-4274-accf-c4f89b655592\") " Jul 17 18:35:21.040630 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:21.040607 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2afaa3-8e69-4274-accf-c4f89b655592-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c2afaa3-8e69-4274-accf-c4f89b655592" (UID: "2c2afaa3-8e69-4274-accf-c4f89b655592"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:35:21.140927 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:21.140853 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c2afaa3-8e69-4274-accf-c4f89b655592-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:35:21.830094 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:21.830051 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq" event={"ID":"2c2afaa3-8e69-4274-accf-c4f89b655592","Type":"ContainerDied","Data":"16ba41e6ebf7c90907eb1458aa297fc895a2c454a8189ee0b1019a7a94b6310c"} Jul 17 18:35:21.830094 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:21.830076 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq" Jul 17 18:35:21.830463 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:21.830120 2572 scope.go:117] "RemoveContainer" containerID="87b0578e64cf6f63d282bcaa6120e1558982da8556974548304fdd29f19672c7" Jul 17 18:35:21.845917 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:21.845883 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq"] Jul 17 18:35:21.853579 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:21.853557 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-595sq"] Jul 17 18:35:23.224384 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:23.224346 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c2afaa3-8e69-4274-accf-c4f89b655592" path="/var/lib/kubelet/pods/2c2afaa3-8e69-4274-accf-c4f89b655592/volumes" Jul 17 18:35:24.954267 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954233 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8"] Jul 17 18:35:24.954637 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954557 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8b69d0f-3ec5-41df-b11f-191aff5cde7c" containerName="revoke-keys" Jul 17 18:35:24.954637 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954568 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8b69d0f-3ec5-41df-b11f-191aff5cde7c" containerName="revoke-keys" Jul 17 18:35:24.954637 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954577 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8b69d0f-3ec5-41df-b11f-191aff5cde7c" containerName="revoke-keys" Jul 17 18:35:24.954637 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954582 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8b69d0f-3ec5-41df-b11f-191aff5cde7c" containerName="revoke-keys" Jul 17 18:35:24.954637 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954591 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8b69d0f-3ec5-41df-b11f-191aff5cde7c" containerName="revoke-keys" Jul 17 18:35:24.954637 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954597 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8b69d0f-3ec5-41df-b11f-191aff5cde7c" containerName="revoke-keys" Jul 17 18:35:24.954637 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954604 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2afaa3-8e69-4274-accf-c4f89b655592" containerName="revoke-keys" Jul 17 18:35:24.954637 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954609 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2afaa3-8e69-4274-accf-c4f89b655592" containerName="revoke-keys" Jul 17 18:35:24.954637 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954618 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2afaa3-8e69-4274-accf-c4f89b655592" containerName="revoke-keys" Jul 17 18:35:24.954637 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954623 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2afaa3-8e69-4274-accf-c4f89b655592" containerName="revoke-keys" Jul 17 18:35:24.954637 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954633 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2afaa3-8e69-4274-accf-c4f89b655592" containerName="revoke-keys" Jul 17 18:35:24.954637 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954638 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2afaa3-8e69-4274-accf-c4f89b655592" containerName="revoke-keys" Jul 17 18:35:24.955003 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954683 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8b69d0f-3ec5-41df-b11f-191aff5cde7c" containerName="revoke-keys" Jul 17 18:35:24.955003 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954690 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8b69d0f-3ec5-41df-b11f-191aff5cde7c" containerName="revoke-keys" Jul 17 18:35:24.955003 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954696 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8b69d0f-3ec5-41df-b11f-191aff5cde7c" containerName="revoke-keys" Jul 17 18:35:24.955003 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954702 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2afaa3-8e69-4274-accf-c4f89b655592" containerName="revoke-keys" Jul 17 18:35:24.955003 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954709 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2afaa3-8e69-4274-accf-c4f89b655592" containerName="revoke-keys" Jul 17 18:35:24.955003 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.954715 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2afaa3-8e69-4274-accf-c4f89b655592" containerName="revoke-keys" Jul 17 18:35:24.958878 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.958862 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8" Jul 17 18:35:24.961399 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.961376 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:35:24.961517 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.961407 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:35:24.964690 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:24.964664 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8"] Jul 17 18:35:25.074752 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:25.074722 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc023ff2-12f9-4000-9e63-4790172656a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8\" (UID: \"bc023ff2-12f9-4000-9e63-4790172656a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8" Jul 17 18:35:25.175343 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:25.175315 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc023ff2-12f9-4000-9e63-4790172656a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8\" (UID: \"bc023ff2-12f9-4000-9e63-4790172656a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8" Jul 17 18:35:25.175950 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:25.175933 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc023ff2-12f9-4000-9e63-4790172656a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8\" (UID: \"bc023ff2-12f9-4000-9e63-4790172656a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8" Jul 17 18:35:25.271765 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:25.271692 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8" Jul 17 18:35:25.393990 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:25.393966 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8"] Jul 17 18:35:25.396242 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:35:25.396217 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc023ff2_12f9_4000_9e63_4790172656a1.slice/crio-db660facdbe41dc23c44a8840220845b27703dcfd0b6371d1afa4f8c9295bca6 WatchSource:0}: Error finding container db660facdbe41dc23c44a8840220845b27703dcfd0b6371d1afa4f8c9295bca6: Status 404 returned error can't find the container with id db660facdbe41dc23c44a8840220845b27703dcfd0b6371d1afa4f8c9295bca6 Jul 17 18:35:25.848979 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:25.848941 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8" event={"ID":"bc023ff2-12f9-4000-9e63-4790172656a1","Type":"ContainerStarted","Data":"693c989aaa1c820c1cb3a8d5d90dedac9b77812ebeab17e3ed3ceaf4ddb8eff3"} Jul 17 18:35:25.849151 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:25.849004 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8" event={"ID":"bc023ff2-12f9-4000-9e63-4790172656a1","Type":"ContainerStarted","Data":"db660facdbe41dc23c44a8840220845b27703dcfd0b6371d1afa4f8c9295bca6"} Jul 17 18:35:25.865660 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:25.865614 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8" podStartSLOduration=1.8656011989999999 podStartE2EDuration="1.865601199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:35:25.86433545 +0000 UTC m=+2321.273763076" watchObservedRunningTime="2026-07-17 18:35:25.865601199 +0000 UTC m=+2321.275028805" Jul 17 18:35:26.859931 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:26.859888 2572 generic.go:356] "Generic (PLEG): container finished" podID="bc023ff2-12f9-4000-9e63-4790172656a1" containerID="693c989aaa1c820c1cb3a8d5d90dedac9b77812ebeab17e3ed3ceaf4ddb8eff3" exitCode=6 Jul 17 18:35:26.860414 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:26.859970 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8" event={"ID":"bc023ff2-12f9-4000-9e63-4790172656a1","Type":"ContainerDied","Data":"693c989aaa1c820c1cb3a8d5d90dedac9b77812ebeab17e3ed3ceaf4ddb8eff3"} Jul 17 18:35:26.860414 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:26.860164 2572 scope.go:117] "RemoveContainer" containerID="693c989aaa1c820c1cb3a8d5d90dedac9b77812ebeab17e3ed3ceaf4ddb8eff3" Jul 17 18:35:27.865970 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:27.865938 2572 generic.go:356] "Generic (PLEG): container finished" podID="bc023ff2-12f9-4000-9e63-4790172656a1" containerID="c919bcc478f8befad13bfc381fc281088a0ce589b1e45245ebc257a87145090c" exitCode=6 Jul 17 18:35:27.866473 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:27.865990 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8" event={"ID":"bc023ff2-12f9-4000-9e63-4790172656a1","Type":"ContainerDied","Data":"c919bcc478f8befad13bfc381fc281088a0ce589b1e45245ebc257a87145090c"} Jul 17 18:35:27.866473 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:27.866069 2572 scope.go:117] "RemoveContainer" containerID="693c989aaa1c820c1cb3a8d5d90dedac9b77812ebeab17e3ed3ceaf4ddb8eff3" Jul 17 18:35:27.866473 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:27.866210 2572 scope.go:117] "RemoveContainer" containerID="c919bcc478f8befad13bfc381fc281088a0ce589b1e45245ebc257a87145090c" Jul 17 18:35:27.866473 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:35:27.866431 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8_odh-ai-gateway-infra(bc023ff2-12f9-4000-9e63-4790172656a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8" podUID="bc023ff2-12f9-4000-9e63-4790172656a1" Jul 17 18:35:28.872201 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:28.872176 2572 scope.go:117] "RemoveContainer" containerID="c919bcc478f8befad13bfc381fc281088a0ce589b1e45245ebc257a87145090c" Jul 17 18:35:28.872591 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:35:28.872336 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8_odh-ai-gateway-infra(bc023ff2-12f9-4000-9e63-4790172656a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8" podUID="bc023ff2-12f9-4000-9e63-4790172656a1" Jul 17 18:35:42.220307 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:42.220220 2572 scope.go:117] "RemoveContainer" containerID="c919bcc478f8befad13bfc381fc281088a0ce589b1e45245ebc257a87145090c" Jul 17 18:35:42.931967 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:42.931935 2572 generic.go:356] "Generic (PLEG): container finished" podID="bc023ff2-12f9-4000-9e63-4790172656a1" containerID="e17ba87776cca761d0e9c7306a92208adedf6861f07af1f05e2a8d30f4ac3c0b" exitCode=6 Jul 17 18:35:42.932128 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:42.932001 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8" event={"ID":"bc023ff2-12f9-4000-9e63-4790172656a1","Type":"ContainerDied","Data":"e17ba87776cca761d0e9c7306a92208adedf6861f07af1f05e2a8d30f4ac3c0b"} Jul 17 18:35:42.932128 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:42.932116 2572 scope.go:117] "RemoveContainer" containerID="c919bcc478f8befad13bfc381fc281088a0ce589b1e45245ebc257a87145090c" Jul 17 18:35:42.932460 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:42.932443 2572 scope.go:117] "RemoveContainer" containerID="e17ba87776cca761d0e9c7306a92208adedf6861f07af1f05e2a8d30f4ac3c0b" Jul 17 18:35:42.932665 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:35:42.932642 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8_odh-ai-gateway-infra(bc023ff2-12f9-4000-9e63-4790172656a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8" podUID="bc023ff2-12f9-4000-9e63-4790172656a1" Jul 17 18:35:43.957253 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:43.957217 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8"] Jul 17 18:35:44.083615 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:44.083593 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8" Jul 17 18:35:44.239315 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:44.239232 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc023ff2-12f9-4000-9e63-4790172656a1-maas-api-service-ca\") pod \"bc023ff2-12f9-4000-9e63-4790172656a1\" (UID: \"bc023ff2-12f9-4000-9e63-4790172656a1\") " Jul 17 18:35:44.239600 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:44.239581 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc023ff2-12f9-4000-9e63-4790172656a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc023ff2-12f9-4000-9e63-4790172656a1" (UID: "bc023ff2-12f9-4000-9e63-4790172656a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:35:44.340258 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:44.340210 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc023ff2-12f9-4000-9e63-4790172656a1-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:35:44.944756 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:44.944724 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8" Jul 17 18:35:44.944756 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:44.944739 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8" event={"ID":"bc023ff2-12f9-4000-9e63-4790172656a1","Type":"ContainerDied","Data":"db660facdbe41dc23c44a8840220845b27703dcfd0b6371d1afa4f8c9295bca6"} Jul 17 18:35:44.944987 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:44.944798 2572 scope.go:117] "RemoveContainer" containerID="e17ba87776cca761d0e9c7306a92208adedf6861f07af1f05e2a8d30f4ac3c0b" Jul 17 18:35:44.963975 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:44.963951 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8"] Jul 17 18:35:44.970090 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:44.970066 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-xdpz8"] Jul 17 18:35:45.224701 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:45.224625 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc023ff2-12f9-4000-9e63-4790172656a1" path="/var/lib/kubelet/pods/bc023ff2-12f9-4000-9e63-4790172656a1/volumes" Jul 17 18:35:53.144302 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.144265 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh"] Jul 17 18:35:53.144793 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.144558 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc023ff2-12f9-4000-9e63-4790172656a1" containerName="revoke-keys" Jul 17 18:35:53.144793 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.144569 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc023ff2-12f9-4000-9e63-4790172656a1" containerName="revoke-keys" Jul 17 18:35:53.144793 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.144577 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc023ff2-12f9-4000-9e63-4790172656a1" containerName="revoke-keys" Jul 17 18:35:53.144793 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.144583 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc023ff2-12f9-4000-9e63-4790172656a1" containerName="revoke-keys" Jul 17 18:35:53.144793 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.144632 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc023ff2-12f9-4000-9e63-4790172656a1" containerName="revoke-keys" Jul 17 18:35:53.144793 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.144643 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc023ff2-12f9-4000-9e63-4790172656a1" containerName="revoke-keys" Jul 17 18:35:53.147229 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.147210 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh" Jul 17 18:35:53.150049 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.150004 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:35:53.150049 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.150041 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:35:53.155114 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.155093 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh"] Jul 17 18:35:53.317658 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.317628 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924c46a1-70d5-46f0-bfb7-e83df50efd3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-v97zh\" (UID: \"924c46a1-70d5-46f0-bfb7-e83df50efd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh" Jul 17 18:35:53.418622 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.418539 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924c46a1-70d5-46f0-bfb7-e83df50efd3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-v97zh\" (UID: \"924c46a1-70d5-46f0-bfb7-e83df50efd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh" Jul 17 18:35:53.419228 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.419207 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924c46a1-70d5-46f0-bfb7-e83df50efd3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-v97zh\" (UID: \"924c46a1-70d5-46f0-bfb7-e83df50efd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh" Jul 17 18:35:53.457540 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.457518 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh" Jul 17 18:35:53.581096 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.581072 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh"] Jul 17 18:35:53.583307 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:35:53.583276 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924c46a1_70d5_46f0_bfb7_e83df50efd3c.slice/crio-3f65d766317b8bd2b159b5c4581a53975bb63291c56f830a16f9fbdf0d79a5f7 WatchSource:0}: Error finding container 3f65d766317b8bd2b159b5c4581a53975bb63291c56f830a16f9fbdf0d79a5f7: Status 404 returned error can't find the container with id 3f65d766317b8bd2b159b5c4581a53975bb63291c56f830a16f9fbdf0d79a5f7 Jul 17 18:35:53.982525 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.982492 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh" event={"ID":"924c46a1-70d5-46f0-bfb7-e83df50efd3c","Type":"ContainerStarted","Data":"6ec2b1c7a380a43ff814c2d5c4a31db7bcff08fdcb3b59965e00322c816cf6d0"} Jul 17 18:35:53.982681 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.982544 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh" event={"ID":"924c46a1-70d5-46f0-bfb7-e83df50efd3c","Type":"ContainerStarted","Data":"3f65d766317b8bd2b159b5c4581a53975bb63291c56f830a16f9fbdf0d79a5f7"} Jul 17 18:35:53.998550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:53.998505 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh" podStartSLOduration=0.998489419 podStartE2EDuration="998.489419ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:35:53.996863998 +0000 UTC m=+2349.406291607" watchObservedRunningTime="2026-07-17 18:35:53.998489419 +0000 UTC m=+2349.407917061" Jul 17 18:35:54.987790 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:54.987758 2572 generic.go:356] "Generic (PLEG): container finished" podID="924c46a1-70d5-46f0-bfb7-e83df50efd3c" containerID="6ec2b1c7a380a43ff814c2d5c4a31db7bcff08fdcb3b59965e00322c816cf6d0" exitCode=6 Jul 17 18:35:54.988197 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:54.987798 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh" event={"ID":"924c46a1-70d5-46f0-bfb7-e83df50efd3c","Type":"ContainerDied","Data":"6ec2b1c7a380a43ff814c2d5c4a31db7bcff08fdcb3b59965e00322c816cf6d0"} Jul 17 18:35:54.988197 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:54.988026 2572 scope.go:117] "RemoveContainer" containerID="6ec2b1c7a380a43ff814c2d5c4a31db7bcff08fdcb3b59965e00322c816cf6d0" Jul 17 18:35:55.995704 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:55.995617 2572 generic.go:356] "Generic (PLEG): container finished" podID="924c46a1-70d5-46f0-bfb7-e83df50efd3c" containerID="7370ffa6a67ed12d2e8338a976fb5d44110216792b07d08819be12fcd8a11a19" exitCode=6 Jul 17 18:35:55.996176 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:55.995709 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh" event={"ID":"924c46a1-70d5-46f0-bfb7-e83df50efd3c","Type":"ContainerDied","Data":"7370ffa6a67ed12d2e8338a976fb5d44110216792b07d08819be12fcd8a11a19"} Jul 17 18:35:55.996176 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:55.995781 2572 scope.go:117] "RemoveContainer" containerID="6ec2b1c7a380a43ff814c2d5c4a31db7bcff08fdcb3b59965e00322c816cf6d0" Jul 17 18:35:55.996176 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:55.995900 2572 scope.go:117] "RemoveContainer" containerID="7370ffa6a67ed12d2e8338a976fb5d44110216792b07d08819be12fcd8a11a19" Jul 17 18:35:55.996176 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:35:55.996109 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-v97zh_odh-ai-gateway-infra(924c46a1-70d5-46f0-bfb7-e83df50efd3c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh" podUID="924c46a1-70d5-46f0-bfb7-e83df50efd3c" Jul 17 18:35:57.002053 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:35:57.002000 2572 scope.go:117] "RemoveContainer" containerID="7370ffa6a67ed12d2e8338a976fb5d44110216792b07d08819be12fcd8a11a19" Jul 17 18:35:57.002429 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:35:57.002202 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-v97zh_odh-ai-gateway-infra(924c46a1-70d5-46f0-bfb7-e83df50efd3c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh" podUID="924c46a1-70d5-46f0-bfb7-e83df50efd3c" Jul 17 18:36:09.220116 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:09.220086 2572 scope.go:117] "RemoveContainer" containerID="7370ffa6a67ed12d2e8338a976fb5d44110216792b07d08819be12fcd8a11a19" Jul 17 18:36:10.057617 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:10.057587 2572 generic.go:356] "Generic (PLEG): container finished" podID="924c46a1-70d5-46f0-bfb7-e83df50efd3c" containerID="47c7cdced0e9c482532261392f57cc8938ccb10269e008834f641455b31f7fb1" exitCode=6 Jul 17 18:36:10.057809 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:10.057661 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh" event={"ID":"924c46a1-70d5-46f0-bfb7-e83df50efd3c","Type":"ContainerDied","Data":"47c7cdced0e9c482532261392f57cc8938ccb10269e008834f641455b31f7fb1"} Jul 17 18:36:10.057809 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:10.057735 2572 scope.go:117] "RemoveContainer" containerID="7370ffa6a67ed12d2e8338a976fb5d44110216792b07d08819be12fcd8a11a19" Jul 17 18:36:10.058035 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:10.058003 2572 scope.go:117] "RemoveContainer" containerID="47c7cdced0e9c482532261392f57cc8938ccb10269e008834f641455b31f7fb1" Jul 17 18:36:10.058252 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:36:10.058230 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-v97zh_odh-ai-gateway-infra(924c46a1-70d5-46f0-bfb7-e83df50efd3c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh" podUID="924c46a1-70d5-46f0-bfb7-e83df50efd3c" Jul 17 18:36:10.624293 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:10.624255 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9"] Jul 17 18:36:10.624670 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:10.624568 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc023ff2-12f9-4000-9e63-4790172656a1" containerName="revoke-keys" Jul 17 18:36:10.624670 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:10.624579 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc023ff2-12f9-4000-9e63-4790172656a1" containerName="revoke-keys" Jul 17 18:36:10.624670 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:10.624646 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc023ff2-12f9-4000-9e63-4790172656a1" containerName="revoke-keys" Jul 17 18:36:10.628748 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:10.628732 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9" Jul 17 18:36:10.636038 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:10.635965 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9"] Jul 17 18:36:10.647725 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:10.647704 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf9e7b4-0592-4b93-b269-59fe657c74bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9\" (UID: \"4cf9e7b4-0592-4b93-b269-59fe657c74bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9" Jul 17 18:36:10.748359 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:10.748324 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf9e7b4-0592-4b93-b269-59fe657c74bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9\" (UID: \"4cf9e7b4-0592-4b93-b269-59fe657c74bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9" Jul 17 18:36:10.748957 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:10.748937 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf9e7b4-0592-4b93-b269-59fe657c74bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9\" (UID: \"4cf9e7b4-0592-4b93-b269-59fe657c74bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9" Jul 17 18:36:10.940450 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:10.940365 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9" Jul 17 18:36:11.064781 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:11.064745 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9"] Jul 17 18:36:11.066689 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:36:11.066659 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf9e7b4_0592_4b93_b269_59fe657c74bb.slice/crio-e2dcd8c7353ce925726ded6bb76b7ffb809ac905b77d025293db847ddfce8252 WatchSource:0}: Error finding container e2dcd8c7353ce925726ded6bb76b7ffb809ac905b77d025293db847ddfce8252: Status 404 returned error can't find the container with id e2dcd8c7353ce925726ded6bb76b7ffb809ac905b77d025293db847ddfce8252 Jul 17 18:36:11.081825 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:11.081802 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh"] Jul 17 18:36:11.206308 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:11.206289 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh" Jul 17 18:36:11.251879 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:11.251854 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924c46a1-70d5-46f0-bfb7-e83df50efd3c-maas-api-service-ca\") pod \"924c46a1-70d5-46f0-bfb7-e83df50efd3c\" (UID: \"924c46a1-70d5-46f0-bfb7-e83df50efd3c\") " Jul 17 18:36:11.252167 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:11.252148 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924c46a1-70d5-46f0-bfb7-e83df50efd3c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "924c46a1-70d5-46f0-bfb7-e83df50efd3c" (UID: "924c46a1-70d5-46f0-bfb7-e83df50efd3c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:36:11.353090 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:11.353059 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/924c46a1-70d5-46f0-bfb7-e83df50efd3c-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:36:12.070080 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:12.070049 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh" Jul 17 18:36:12.070561 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:12.070076 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh" event={"ID":"924c46a1-70d5-46f0-bfb7-e83df50efd3c","Type":"ContainerDied","Data":"3f65d766317b8bd2b159b5c4581a53975bb63291c56f830a16f9fbdf0d79a5f7"} Jul 17 18:36:12.070561 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:12.070144 2572 scope.go:117] "RemoveContainer" containerID="47c7cdced0e9c482532261392f57cc8938ccb10269e008834f641455b31f7fb1" Jul 17 18:36:12.071593 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:12.071568 2572 generic.go:356] "Generic (PLEG): container finished" podID="4cf9e7b4-0592-4b93-b269-59fe657c74bb" containerID="84dbaf62441fda3c4d2601833e6f8f80174702df5585413f204f213099d4f0be" exitCode=7 Jul 17 18:36:12.071684 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:12.071628 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9" event={"ID":"4cf9e7b4-0592-4b93-b269-59fe657c74bb","Type":"ContainerDied","Data":"84dbaf62441fda3c4d2601833e6f8f80174702df5585413f204f213099d4f0be"} Jul 17 18:36:12.071684 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:12.071671 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9" event={"ID":"4cf9e7b4-0592-4b93-b269-59fe657c74bb","Type":"ContainerStarted","Data":"e2dcd8c7353ce925726ded6bb76b7ffb809ac905b77d025293db847ddfce8252"} Jul 17 18:36:12.071854 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:12.071841 2572 scope.go:117] "RemoveContainer" containerID="84dbaf62441fda3c4d2601833e6f8f80174702df5585413f204f213099d4f0be" Jul 17 18:36:12.106613 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:12.106588 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh"] Jul 17 18:36:12.110908 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:12.110887 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-v97zh"] Jul 17 18:36:13.078556 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:13.078520 2572 generic.go:356] "Generic (PLEG): container finished" podID="4cf9e7b4-0592-4b93-b269-59fe657c74bb" containerID="26ca12f0484a7e87a10a268662e959a97b55a8fdd22ff49ab168c7d07e845b16" exitCode=7 Jul 17 18:36:13.079055 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:13.078576 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9" event={"ID":"4cf9e7b4-0592-4b93-b269-59fe657c74bb","Type":"ContainerDied","Data":"26ca12f0484a7e87a10a268662e959a97b55a8fdd22ff49ab168c7d07e845b16"} Jul 17 18:36:13.079055 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:13.078633 2572 scope.go:117] "RemoveContainer" containerID="84dbaf62441fda3c4d2601833e6f8f80174702df5585413f204f213099d4f0be" Jul 17 18:36:13.079055 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:13.078834 2572 scope.go:117] "RemoveContainer" containerID="26ca12f0484a7e87a10a268662e959a97b55a8fdd22ff49ab168c7d07e845b16" Jul 17 18:36:13.079231 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:36:13.079084 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9_odh-ai-gateway-infra(4cf9e7b4-0592-4b93-b269-59fe657c74bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9" podUID="4cf9e7b4-0592-4b93-b269-59fe657c74bb" Jul 17 18:36:13.224703 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:13.224672 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="924c46a1-70d5-46f0-bfb7-e83df50efd3c" path="/var/lib/kubelet/pods/924c46a1-70d5-46f0-bfb7-e83df50efd3c/volumes" Jul 17 18:36:14.084811 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:14.084784 2572 scope.go:117] "RemoveContainer" containerID="26ca12f0484a7e87a10a268662e959a97b55a8fdd22ff49ab168c7d07e845b16" Jul 17 18:36:14.085218 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:36:14.084961 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9_odh-ai-gateway-infra(4cf9e7b4-0592-4b93-b269-59fe657c74bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9" podUID="4cf9e7b4-0592-4b93-b269-59fe657c74bb" Jul 17 18:36:26.219851 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:26.219819 2572 scope.go:117] "RemoveContainer" containerID="26ca12f0484a7e87a10a268662e959a97b55a8fdd22ff49ab168c7d07e845b16" Jul 17 18:36:27.138141 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:27.138112 2572 generic.go:356] "Generic (PLEG): container finished" podID="4cf9e7b4-0592-4b93-b269-59fe657c74bb" containerID="e12a79f2ec2a77e56040173302671871d1662df65ac8af2fced40740d115c2b3" exitCode=6 Jul 17 18:36:27.138291 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:27.138188 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9" event={"ID":"4cf9e7b4-0592-4b93-b269-59fe657c74bb","Type":"ContainerDied","Data":"e12a79f2ec2a77e56040173302671871d1662df65ac8af2fced40740d115c2b3"} Jul 17 18:36:27.138291 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:27.138249 2572 scope.go:117] "RemoveContainer" containerID="26ca12f0484a7e87a10a268662e959a97b55a8fdd22ff49ab168c7d07e845b16" Jul 17 18:36:27.138559 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:27.138544 2572 scope.go:117] "RemoveContainer" containerID="e12a79f2ec2a77e56040173302671871d1662df65ac8af2fced40740d115c2b3" Jul 17 18:36:27.138769 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:36:27.138746 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9_odh-ai-gateway-infra(4cf9e7b4-0592-4b93-b269-59fe657c74bb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9" podUID="4cf9e7b4-0592-4b93-b269-59fe657c74bb" Jul 17 18:36:28.163813 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:28.163760 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9"] Jul 17 18:36:28.313037 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:28.312996 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9" Jul 17 18:36:28.389541 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:28.389513 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf9e7b4-0592-4b93-b269-59fe657c74bb-maas-api-service-ca\") pod \"4cf9e7b4-0592-4b93-b269-59fe657c74bb\" (UID: \"4cf9e7b4-0592-4b93-b269-59fe657c74bb\") " Jul 17 18:36:28.389843 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:28.389824 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf9e7b4-0592-4b93-b269-59fe657c74bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4cf9e7b4-0592-4b93-b269-59fe657c74bb" (UID: "4cf9e7b4-0592-4b93-b269-59fe657c74bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:36:28.490718 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:28.490641 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf9e7b4-0592-4b93-b269-59fe657c74bb-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:36:29.148721 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:29.148680 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9" event={"ID":"4cf9e7b4-0592-4b93-b269-59fe657c74bb","Type":"ContainerDied","Data":"e2dcd8c7353ce925726ded6bb76b7ffb809ac905b77d025293db847ddfce8252"} Jul 17 18:36:29.148721 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:29.148698 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9" Jul 17 18:36:29.148957 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:29.148740 2572 scope.go:117] "RemoveContainer" containerID="e12a79f2ec2a77e56040173302671871d1662df65ac8af2fced40740d115c2b3" Jul 17 18:36:29.170935 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:29.170907 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9"] Jul 17 18:36:29.180668 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:29.180646 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-5hdp9"] Jul 17 18:36:29.224096 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:29.224069 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cf9e7b4-0592-4b93-b269-59fe657c74bb" path="/var/lib/kubelet/pods/4cf9e7b4-0592-4b93-b269-59fe657c74bb/volumes" Jul 17 18:36:37.141767 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.141730 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp"] Jul 17 18:36:37.142208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.142053 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf9e7b4-0592-4b93-b269-59fe657c74bb" containerName="revoke-keys" Jul 17 18:36:37.142208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.142065 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf9e7b4-0592-4b93-b269-59fe657c74bb" containerName="revoke-keys" Jul 17 18:36:37.142208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.142077 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="924c46a1-70d5-46f0-bfb7-e83df50efd3c" containerName="revoke-keys" Jul 17 18:36:37.142208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.142082 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="924c46a1-70d5-46f0-bfb7-e83df50efd3c" containerName="revoke-keys" Jul 17 18:36:37.142208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.142098 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf9e7b4-0592-4b93-b269-59fe657c74bb" containerName="revoke-keys" Jul 17 18:36:37.142208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.142105 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf9e7b4-0592-4b93-b269-59fe657c74bb" containerName="revoke-keys" Jul 17 18:36:37.142208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.142128 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="924c46a1-70d5-46f0-bfb7-e83df50efd3c" containerName="revoke-keys" Jul 17 18:36:37.142208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.142134 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="924c46a1-70d5-46f0-bfb7-e83df50efd3c" containerName="revoke-keys" Jul 17 18:36:37.142208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.142184 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="924c46a1-70d5-46f0-bfb7-e83df50efd3c" containerName="revoke-keys" Jul 17 18:36:37.142208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.142192 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="924c46a1-70d5-46f0-bfb7-e83df50efd3c" containerName="revoke-keys" Jul 17 18:36:37.142208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.142198 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="924c46a1-70d5-46f0-bfb7-e83df50efd3c" containerName="revoke-keys" Jul 17 18:36:37.142208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.142203 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf9e7b4-0592-4b93-b269-59fe657c74bb" containerName="revoke-keys" Jul 17 18:36:37.142208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.142209 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf9e7b4-0592-4b93-b269-59fe657c74bb" containerName="revoke-keys" Jul 17 18:36:37.142208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.142215 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cf9e7b4-0592-4b93-b269-59fe657c74bb" containerName="revoke-keys" Jul 17 18:36:37.146328 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.146312 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp" Jul 17 18:36:37.148988 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.148965 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:36:37.149135 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.148965 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:36:37.152665 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.152629 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp"] Jul 17 18:36:37.266161 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.266133 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de83e354-3b81-45a3-b484-5615d4b9038f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp\" (UID: \"de83e354-3b81-45a3-b484-5615d4b9038f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp" Jul 17 18:36:37.366837 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.366809 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de83e354-3b81-45a3-b484-5615d4b9038f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp\" (UID: \"de83e354-3b81-45a3-b484-5615d4b9038f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp" Jul 17 18:36:37.367421 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.367404 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de83e354-3b81-45a3-b484-5615d4b9038f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp\" (UID: \"de83e354-3b81-45a3-b484-5615d4b9038f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp" Jul 17 18:36:37.457063 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.456981 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp" Jul 17 18:36:37.582488 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:37.582462 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp"] Jul 17 18:36:37.584323 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:36:37.584291 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde83e354_3b81_45a3_b484_5615d4b9038f.slice/crio-1c22cd84433ffe0c865e4d7ae1bd2069d0ad2c6147e557f1d5b89fcb503dcf83 WatchSource:0}: Error finding container 1c22cd84433ffe0c865e4d7ae1bd2069d0ad2c6147e557f1d5b89fcb503dcf83: Status 404 returned error can't find the container with id 1c22cd84433ffe0c865e4d7ae1bd2069d0ad2c6147e557f1d5b89fcb503dcf83 Jul 17 18:36:38.188296 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:38.188264 2572 generic.go:356] "Generic (PLEG): container finished" podID="de83e354-3b81-45a3-b484-5615d4b9038f" containerID="02eba2629d299382d07355ceef91c660bf31be7df4c4bfa11f46b71455b771d8" exitCode=6 Jul 17 18:36:38.188693 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:38.188334 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp" event={"ID":"de83e354-3b81-45a3-b484-5615d4b9038f","Type":"ContainerDied","Data":"02eba2629d299382d07355ceef91c660bf31be7df4c4bfa11f46b71455b771d8"} Jul 17 18:36:38.188693 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:38.188390 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp" event={"ID":"de83e354-3b81-45a3-b484-5615d4b9038f","Type":"ContainerStarted","Data":"1c22cd84433ffe0c865e4d7ae1bd2069d0ad2c6147e557f1d5b89fcb503dcf83"} Jul 17 18:36:38.188693 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:38.188574 2572 scope.go:117] "RemoveContainer" containerID="02eba2629d299382d07355ceef91c660bf31be7df4c4bfa11f46b71455b771d8" Jul 17 18:36:39.193924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:39.193841 2572 generic.go:356] "Generic (PLEG): container finished" podID="de83e354-3b81-45a3-b484-5615d4b9038f" containerID="21d19e31a499597d87c8437e6f5e095c1ae6696f704d0bae406b85a883c21d3c" exitCode=6 Jul 17 18:36:39.194318 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:39.193919 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp" event={"ID":"de83e354-3b81-45a3-b484-5615d4b9038f","Type":"ContainerDied","Data":"21d19e31a499597d87c8437e6f5e095c1ae6696f704d0bae406b85a883c21d3c"} Jul 17 18:36:39.194318 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:39.193982 2572 scope.go:117] "RemoveContainer" containerID="02eba2629d299382d07355ceef91c660bf31be7df4c4bfa11f46b71455b771d8" Jul 17 18:36:39.194318 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:39.194231 2572 scope.go:117] "RemoveContainer" containerID="21d19e31a499597d87c8437e6f5e095c1ae6696f704d0bae406b85a883c21d3c" Jul 17 18:36:39.194564 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:36:39.194508 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp_odh-ai-gateway-infra(de83e354-3b81-45a3-b484-5615d4b9038f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp" podUID="de83e354-3b81-45a3-b484-5615d4b9038f" Jul 17 18:36:40.200574 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:40.200543 2572 scope.go:117] "RemoveContainer" containerID="21d19e31a499597d87c8437e6f5e095c1ae6696f704d0bae406b85a883c21d3c" Jul 17 18:36:40.201066 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:36:40.200728 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp_odh-ai-gateway-infra(de83e354-3b81-45a3-b484-5615d4b9038f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp" podUID="de83e354-3b81-45a3-b484-5615d4b9038f" Jul 17 18:36:53.219489 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:53.219447 2572 scope.go:117] "RemoveContainer" containerID="21d19e31a499597d87c8437e6f5e095c1ae6696f704d0bae406b85a883c21d3c" Jul 17 18:36:54.257558 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:54.257479 2572 generic.go:356] "Generic (PLEG): container finished" podID="de83e354-3b81-45a3-b484-5615d4b9038f" containerID="3a121b600b27231ea26625d251f7179c118828712a04834c580d504603fffebb" exitCode=6 Jul 17 18:36:54.257915 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:54.257553 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp" event={"ID":"de83e354-3b81-45a3-b484-5615d4b9038f","Type":"ContainerDied","Data":"3a121b600b27231ea26625d251f7179c118828712a04834c580d504603fffebb"} Jul 17 18:36:54.257915 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:54.257614 2572 scope.go:117] "RemoveContainer" containerID="21d19e31a499597d87c8437e6f5e095c1ae6696f704d0bae406b85a883c21d3c" Jul 17 18:36:54.258045 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:54.257953 2572 scope.go:117] "RemoveContainer" containerID="3a121b600b27231ea26625d251f7179c118828712a04834c580d504603fffebb" Jul 17 18:36:54.258190 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:36:54.258167 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp_odh-ai-gateway-infra(de83e354-3b81-45a3-b484-5615d4b9038f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp" podUID="de83e354-3b81-45a3-b484-5615d4b9038f" Jul 17 18:36:55.283730 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:55.283699 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp"] Jul 17 18:36:55.410287 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:55.410263 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp" Jul 17 18:36:55.520687 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:55.520654 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de83e354-3b81-45a3-b484-5615d4b9038f-maas-api-service-ca\") pod \"de83e354-3b81-45a3-b484-5615d4b9038f\" (UID: \"de83e354-3b81-45a3-b484-5615d4b9038f\") " Jul 17 18:36:55.521057 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:55.521000 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de83e354-3b81-45a3-b484-5615d4b9038f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "de83e354-3b81-45a3-b484-5615d4b9038f" (UID: "de83e354-3b81-45a3-b484-5615d4b9038f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:36:55.622052 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:55.621964 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de83e354-3b81-45a3-b484-5615d4b9038f-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:36:56.267963 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:56.267936 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp" Jul 17 18:36:56.268184 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:56.267939 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp" event={"ID":"de83e354-3b81-45a3-b484-5615d4b9038f","Type":"ContainerDied","Data":"1c22cd84433ffe0c865e4d7ae1bd2069d0ad2c6147e557f1d5b89fcb503dcf83"} Jul 17 18:36:56.268184 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:56.268061 2572 scope.go:117] "RemoveContainer" containerID="3a121b600b27231ea26625d251f7179c118828712a04834c580d504603fffebb" Jul 17 18:36:56.288087 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:56.288063 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp"] Jul 17 18:36:56.294354 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:56.294333 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-zskhp"] Jul 17 18:36:57.224374 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:36:57.224341 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de83e354-3b81-45a3-b484-5615d4b9038f" path="/var/lib/kubelet/pods/de83e354-3b81-45a3-b484-5615d4b9038f/volumes" Jul 17 18:38:25.095808 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.095776 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns"] Jul 17 18:38:25.096251 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.096109 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cf9e7b4-0592-4b93-b269-59fe657c74bb" containerName="revoke-keys" Jul 17 18:38:25.096251 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.096123 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cf9e7b4-0592-4b93-b269-59fe657c74bb" containerName="revoke-keys" Jul 17 18:38:25.096251 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.096134 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de83e354-3b81-45a3-b484-5615d4b9038f" containerName="revoke-keys" Jul 17 18:38:25.096251 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.096139 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="de83e354-3b81-45a3-b484-5615d4b9038f" containerName="revoke-keys" Jul 17 18:38:25.096251 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.096163 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="924c46a1-70d5-46f0-bfb7-e83df50efd3c" containerName="revoke-keys" Jul 17 18:38:25.096251 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.096169 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="924c46a1-70d5-46f0-bfb7-e83df50efd3c" containerName="revoke-keys" Jul 17 18:38:25.096251 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.096186 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de83e354-3b81-45a3-b484-5615d4b9038f" containerName="revoke-keys" Jul 17 18:38:25.096251 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.096192 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="de83e354-3b81-45a3-b484-5615d4b9038f" containerName="revoke-keys" Jul 17 18:38:25.096251 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.096235 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="de83e354-3b81-45a3-b484-5615d4b9038f" containerName="revoke-keys" Jul 17 18:38:25.096251 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.096242 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="de83e354-3b81-45a3-b484-5615d4b9038f" containerName="revoke-keys" Jul 17 18:38:25.096251 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.096249 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="de83e354-3b81-45a3-b484-5615d4b9038f" containerName="revoke-keys" Jul 17 18:38:25.098792 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.098774 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns" Jul 17 18:38:25.101993 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.101969 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:38:25.102133 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.101991 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:38:25.108084 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.108059 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns"] Jul 17 18:38:25.184959 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.184927 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e2218c-1cf2-47ea-9d81-039004147a3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns\" (UID: \"80e2218c-1cf2-47ea-9d81-039004147a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns" Jul 17 18:38:25.285729 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.285686 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e2218c-1cf2-47ea-9d81-039004147a3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns\" (UID: \"80e2218c-1cf2-47ea-9d81-039004147a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns" Jul 17 18:38:25.286350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.286329 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e2218c-1cf2-47ea-9d81-039004147a3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns\" (UID: \"80e2218c-1cf2-47ea-9d81-039004147a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns" Jul 17 18:38:25.410350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.410265 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns" Jul 17 18:38:25.532537 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.532321 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns"] Jul 17 18:38:25.534854 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:38:25.534821 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e2218c_1cf2_47ea_9d81_039004147a3b.slice/crio-c1cf93f53cade4eb16e785ab61aa727bc43bd3148de7c2d45ef7363af78d1629 WatchSource:0}: Error finding container c1cf93f53cade4eb16e785ab61aa727bc43bd3148de7c2d45ef7363af78d1629: Status 404 returned error can't find the container with id c1cf93f53cade4eb16e785ab61aa727bc43bd3148de7c2d45ef7363af78d1629 Jul 17 18:38:25.627911 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:25.627885 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns" event={"ID":"80e2218c-1cf2-47ea-9d81-039004147a3b","Type":"ContainerStarted","Data":"c1cf93f53cade4eb16e785ab61aa727bc43bd3148de7c2d45ef7363af78d1629"} Jul 17 18:38:26.633419 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:26.633386 2572 generic.go:356] "Generic (PLEG): container finished" podID="80e2218c-1cf2-47ea-9d81-039004147a3b" containerID="03d17b04de82b3f50ceb0f6bfb791b210ffd1a9a05320a22384b15c078d55113" exitCode=6 Jul 17 18:38:26.633831 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:26.633431 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns" event={"ID":"80e2218c-1cf2-47ea-9d81-039004147a3b","Type":"ContainerDied","Data":"03d17b04de82b3f50ceb0f6bfb791b210ffd1a9a05320a22384b15c078d55113"} Jul 17 18:38:26.633831 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:26.633637 2572 scope.go:117] "RemoveContainer" containerID="03d17b04de82b3f50ceb0f6bfb791b210ffd1a9a05320a22384b15c078d55113" Jul 17 18:38:27.639244 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:27.639152 2572 generic.go:356] "Generic (PLEG): container finished" podID="80e2218c-1cf2-47ea-9d81-039004147a3b" containerID="5f388409c8862741f2e02e943d48def9fefd752b3e32f5b62767b2296b312022" exitCode=6 Jul 17 18:38:27.639650 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:27.639236 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns" event={"ID":"80e2218c-1cf2-47ea-9d81-039004147a3b","Type":"ContainerDied","Data":"5f388409c8862741f2e02e943d48def9fefd752b3e32f5b62767b2296b312022"} Jul 17 18:38:27.639650 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:27.639295 2572 scope.go:117] "RemoveContainer" containerID="03d17b04de82b3f50ceb0f6bfb791b210ffd1a9a05320a22384b15c078d55113" Jul 17 18:38:27.639650 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:27.639477 2572 scope.go:117] "RemoveContainer" containerID="5f388409c8862741f2e02e943d48def9fefd752b3e32f5b62767b2296b312022" Jul 17 18:38:27.639813 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:38:27.639696 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns_odh-ai-gateway-infra(80e2218c-1cf2-47ea-9d81-039004147a3b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns" podUID="80e2218c-1cf2-47ea-9d81-039004147a3b" Jul 17 18:38:28.644551 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:28.644528 2572 scope.go:117] "RemoveContainer" containerID="5f388409c8862741f2e02e943d48def9fefd752b3e32f5b62767b2296b312022" Jul 17 18:38:28.644931 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:38:28.644722 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns_odh-ai-gateway-infra(80e2218c-1cf2-47ea-9d81-039004147a3b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns" podUID="80e2218c-1cf2-47ea-9d81-039004147a3b" Jul 17 18:38:40.747955 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:40.747871 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn"] Jul 17 18:38:40.748346 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:40.748227 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de83e354-3b81-45a3-b484-5615d4b9038f" containerName="revoke-keys" Jul 17 18:38:40.748346 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:40.748239 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="de83e354-3b81-45a3-b484-5615d4b9038f" containerName="revoke-keys" Jul 17 18:38:40.752260 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:40.752239 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn" Jul 17 18:38:40.761466 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:40.761431 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn"] Jul 17 18:38:40.815419 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:40.815386 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b0ed0d-e2f5-485c-8339-bf41673feed8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn\" (UID: \"71b0ed0d-e2f5-485c-8339-bf41673feed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn" Jul 17 18:38:40.916502 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:40.916454 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b0ed0d-e2f5-485c-8339-bf41673feed8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn\" (UID: \"71b0ed0d-e2f5-485c-8339-bf41673feed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn" Jul 17 18:38:40.917054 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:40.917036 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b0ed0d-e2f5-485c-8339-bf41673feed8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn\" (UID: \"71b0ed0d-e2f5-485c-8339-bf41673feed8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn" Jul 17 18:38:41.065457 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:41.065421 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn" Jul 17 18:38:41.193812 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:41.193766 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn"] Jul 17 18:38:41.196260 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:38:41.196229 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b0ed0d_e2f5_485c_8339_bf41673feed8.slice/crio-33fac34f590233d18b626f55f57120ffd83c47989b8a6b1ad665561ba25adba4 WatchSource:0}: Error finding container 33fac34f590233d18b626f55f57120ffd83c47989b8a6b1ad665561ba25adba4: Status 404 returned error can't find the container with id 33fac34f590233d18b626f55f57120ffd83c47989b8a6b1ad665561ba25adba4 Jul 17 18:38:41.695653 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:41.695546 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn" event={"ID":"71b0ed0d-e2f5-485c-8339-bf41673feed8","Type":"ContainerStarted","Data":"a6e4e68bb96ce266329cdc9e5b971b2243cfee041797aa79b8c4b9582a39def0"} Jul 17 18:38:41.695653 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:41.695615 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn" event={"ID":"71b0ed0d-e2f5-485c-8339-bf41673feed8","Type":"ContainerStarted","Data":"33fac34f590233d18b626f55f57120ffd83c47989b8a6b1ad665561ba25adba4"} Jul 17 18:38:41.714967 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:41.714920 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn" podStartSLOduration=1.7149052980000001 podStartE2EDuration="1.714905298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:38:41.713558179 +0000 UTC m=+2517.122985809" watchObservedRunningTime="2026-07-17 18:38:41.714905298 +0000 UTC m=+2517.124332921" Jul 17 18:38:42.220136 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:42.220106 2572 scope.go:117] "RemoveContainer" containerID="5f388409c8862741f2e02e943d48def9fefd752b3e32f5b62767b2296b312022" Jul 17 18:38:42.701927 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:42.701893 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns" event={"ID":"80e2218c-1cf2-47ea-9d81-039004147a3b","Type":"ContainerStarted","Data":"27a6d1629f66d3b435d30d69a8ee82be33b143e52a3af0771574c987de8cbaa0"} Jul 17 18:38:42.703170 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:42.703143 2572 generic.go:356] "Generic (PLEG): container finished" podID="71b0ed0d-e2f5-485c-8339-bf41673feed8" containerID="a6e4e68bb96ce266329cdc9e5b971b2243cfee041797aa79b8c4b9582a39def0" exitCode=6 Jul 17 18:38:42.703293 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:42.703216 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn" event={"ID":"71b0ed0d-e2f5-485c-8339-bf41673feed8","Type":"ContainerDied","Data":"a6e4e68bb96ce266329cdc9e5b971b2243cfee041797aa79b8c4b9582a39def0"} Jul 17 18:38:42.703361 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:42.703347 2572 scope.go:117] "RemoveContainer" containerID="a6e4e68bb96ce266329cdc9e5b971b2243cfee041797aa79b8c4b9582a39def0" Jul 17 18:38:42.716983 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:42.716942 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns" podStartSLOduration=17.716927412 podStartE2EDuration="17.716927412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:38:42.716559761 +0000 UTC m=+2518.125987368" watchObservedRunningTime="2026-07-17 18:38:42.716927412 +0000 UTC m=+2518.126355018" Jul 17 18:38:43.709697 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:43.709618 2572 generic.go:356] "Generic (PLEG): container finished" podID="80e2218c-1cf2-47ea-9d81-039004147a3b" containerID="27a6d1629f66d3b435d30d69a8ee82be33b143e52a3af0771574c987de8cbaa0" exitCode=6 Jul 17 18:38:43.710149 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:43.709695 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns" event={"ID":"80e2218c-1cf2-47ea-9d81-039004147a3b","Type":"ContainerDied","Data":"27a6d1629f66d3b435d30d69a8ee82be33b143e52a3af0771574c987de8cbaa0"} Jul 17 18:38:43.710149 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:43.709757 2572 scope.go:117] "RemoveContainer" containerID="5f388409c8862741f2e02e943d48def9fefd752b3e32f5b62767b2296b312022" Jul 17 18:38:43.710149 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:43.710091 2572 scope.go:117] "RemoveContainer" containerID="27a6d1629f66d3b435d30d69a8ee82be33b143e52a3af0771574c987de8cbaa0" Jul 17 18:38:43.710390 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:38:43.710319 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns_odh-ai-gateway-infra(80e2218c-1cf2-47ea-9d81-039004147a3b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns" podUID="80e2218c-1cf2-47ea-9d81-039004147a3b" Jul 17 18:38:43.711554 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:43.711535 2572 generic.go:356] "Generic (PLEG): container finished" podID="71b0ed0d-e2f5-485c-8339-bf41673feed8" containerID="2344eae46055ee83f728c2668d9caf9828ea8ab72ec1e4666f0257b3ef0ce0c8" exitCode=6 Jul 17 18:38:43.711666 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:43.711581 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn" event={"ID":"71b0ed0d-e2f5-485c-8339-bf41673feed8","Type":"ContainerDied","Data":"2344eae46055ee83f728c2668d9caf9828ea8ab72ec1e4666f0257b3ef0ce0c8"} Jul 17 18:38:43.711769 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:43.711756 2572 scope.go:117] "RemoveContainer" containerID="2344eae46055ee83f728c2668d9caf9828ea8ab72ec1e4666f0257b3ef0ce0c8" Jul 17 18:38:43.711963 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:38:43.711947 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn_odh-ai-gateway-infra(71b0ed0d-e2f5-485c-8339-bf41673feed8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn" podUID="71b0ed0d-e2f5-485c-8339-bf41673feed8" Jul 17 18:38:43.720427 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:43.720409 2572 scope.go:117] "RemoveContainer" containerID="a6e4e68bb96ce266329cdc9e5b971b2243cfee041797aa79b8c4b9582a39def0" Jul 17 18:38:43.728059 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:43.728035 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns"] Jul 17 18:38:44.719343 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:44.719324 2572 scope.go:117] "RemoveContainer" containerID="2344eae46055ee83f728c2668d9caf9828ea8ab72ec1e4666f0257b3ef0ce0c8" Jul 17 18:38:44.719721 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:38:44.719557 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn_odh-ai-gateway-infra(71b0ed0d-e2f5-485c-8339-bf41673feed8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn" podUID="71b0ed0d-e2f5-485c-8339-bf41673feed8" Jul 17 18:38:44.849142 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:44.849118 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns" Jul 17 18:38:44.943536 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:44.943504 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e2218c-1cf2-47ea-9d81-039004147a3b-maas-api-service-ca\") pod \"80e2218c-1cf2-47ea-9d81-039004147a3b\" (UID: \"80e2218c-1cf2-47ea-9d81-039004147a3b\") " Jul 17 18:38:44.943865 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:44.943844 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e2218c-1cf2-47ea-9d81-039004147a3b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80e2218c-1cf2-47ea-9d81-039004147a3b" (UID: "80e2218c-1cf2-47ea-9d81-039004147a3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:38:45.044535 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:45.044505 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e2218c-1cf2-47ea-9d81-039004147a3b-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:38:45.724793 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:45.724757 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns" event={"ID":"80e2218c-1cf2-47ea-9d81-039004147a3b","Type":"ContainerDied","Data":"c1cf93f53cade4eb16e785ab61aa727bc43bd3148de7c2d45ef7363af78d1629"} Jul 17 18:38:45.725249 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:45.724815 2572 scope.go:117] "RemoveContainer" containerID="27a6d1629f66d3b435d30d69a8ee82be33b143e52a3af0771574c987de8cbaa0" Jul 17 18:38:45.725249 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:45.724770 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns" Jul 17 18:38:45.746795 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:45.746764 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns"] Jul 17 18:38:45.752762 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:45.752738 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qhqns"] Jul 17 18:38:47.224403 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:38:47.224373 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80e2218c-1cf2-47ea-9d81-039004147a3b" path="/var/lib/kubelet/pods/80e2218c-1cf2-47ea-9d81-039004147a3b/volumes" Jul 17 18:39:00.220205 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:00.220177 2572 scope.go:117] "RemoveContainer" containerID="2344eae46055ee83f728c2668d9caf9828ea8ab72ec1e4666f0257b3ef0ce0c8" Jul 17 18:39:00.787165 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:00.787132 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn" event={"ID":"71b0ed0d-e2f5-485c-8339-bf41673feed8","Type":"ContainerStarted","Data":"27833711c6ee38edb7780d47d15d50ea470c04bfc0b604f15ecbafa9d491a2e2"} Jul 17 18:39:01.793164 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:01.793131 2572 generic.go:356] "Generic (PLEG): container finished" podID="71b0ed0d-e2f5-485c-8339-bf41673feed8" containerID="27833711c6ee38edb7780d47d15d50ea470c04bfc0b604f15ecbafa9d491a2e2" exitCode=6 Jul 17 18:39:01.793618 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:01.793207 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn" event={"ID":"71b0ed0d-e2f5-485c-8339-bf41673feed8","Type":"ContainerDied","Data":"27833711c6ee38edb7780d47d15d50ea470c04bfc0b604f15ecbafa9d491a2e2"} Jul 17 18:39:01.793618 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:01.793269 2572 scope.go:117] "RemoveContainer" containerID="2344eae46055ee83f728c2668d9caf9828ea8ab72ec1e4666f0257b3ef0ce0c8" Jul 17 18:39:01.793618 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:01.793523 2572 scope.go:117] "RemoveContainer" containerID="27833711c6ee38edb7780d47d15d50ea470c04bfc0b604f15ecbafa9d491a2e2" Jul 17 18:39:01.793789 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:39:01.793711 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn_odh-ai-gateway-infra(71b0ed0d-e2f5-485c-8339-bf41673feed8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn" podUID="71b0ed0d-e2f5-485c-8339-bf41673feed8" Jul 17 18:39:01.814752 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:01.814724 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn"] Jul 17 18:39:02.929259 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:02.929236 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn" Jul 17 18:39:03.001042 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:03.000996 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b0ed0d-e2f5-485c-8339-bf41673feed8-maas-api-service-ca\") pod \"71b0ed0d-e2f5-485c-8339-bf41673feed8\" (UID: \"71b0ed0d-e2f5-485c-8339-bf41673feed8\") " Jul 17 18:39:03.001358 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:03.001334 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b0ed0d-e2f5-485c-8339-bf41673feed8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "71b0ed0d-e2f5-485c-8339-bf41673feed8" (UID: "71b0ed0d-e2f5-485c-8339-bf41673feed8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:39:03.102508 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:03.102422 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71b0ed0d-e2f5-485c-8339-bf41673feed8-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:39:03.804495 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:03.804456 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn" Jul 17 18:39:03.804676 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:03.804491 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn" event={"ID":"71b0ed0d-e2f5-485c-8339-bf41673feed8","Type":"ContainerDied","Data":"33fac34f590233d18b626f55f57120ffd83c47989b8a6b1ad665561ba25adba4"} Jul 17 18:39:03.804676 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:03.804555 2572 scope.go:117] "RemoveContainer" containerID="27833711c6ee38edb7780d47d15d50ea470c04bfc0b604f15ecbafa9d491a2e2" Jul 17 18:39:03.820210 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:03.820185 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn"] Jul 17 18:39:03.824385 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:03.824362 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-8cjwn"] Jul 17 18:39:05.223926 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:05.223895 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71b0ed0d-e2f5-485c-8339-bf41673feed8" path="/var/lib/kubelet/pods/71b0ed0d-e2f5-485c-8339-bf41673feed8/volumes" Jul 17 18:39:23.395678 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.395645 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn"] Jul 17 18:39:23.396067 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.395914 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b0ed0d-e2f5-485c-8339-bf41673feed8" containerName="revoke-keys" Jul 17 18:39:23.396067 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.395925 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b0ed0d-e2f5-485c-8339-bf41673feed8" containerName="revoke-keys" Jul 17 18:39:23.396067 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.395937 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e2218c-1cf2-47ea-9d81-039004147a3b" containerName="revoke-keys" Jul 17 18:39:23.396067 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.395943 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e2218c-1cf2-47ea-9d81-039004147a3b" containerName="revoke-keys" Jul 17 18:39:23.396067 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.395954 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e2218c-1cf2-47ea-9d81-039004147a3b" containerName="revoke-keys" Jul 17 18:39:23.396067 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.395960 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e2218c-1cf2-47ea-9d81-039004147a3b" containerName="revoke-keys" Jul 17 18:39:23.396067 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.395966 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e2218c-1cf2-47ea-9d81-039004147a3b" containerName="revoke-keys" Jul 17 18:39:23.396067 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.395971 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e2218c-1cf2-47ea-9d81-039004147a3b" containerName="revoke-keys" Jul 17 18:39:23.396067 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.395997 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b0ed0d-e2f5-485c-8339-bf41673feed8" containerName="revoke-keys" Jul 17 18:39:23.396067 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.396002 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b0ed0d-e2f5-485c-8339-bf41673feed8" containerName="revoke-keys" Jul 17 18:39:23.396067 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.396056 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b0ed0d-e2f5-485c-8339-bf41673feed8" containerName="revoke-keys" Jul 17 18:39:23.396067 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.396065 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b0ed0d-e2f5-485c-8339-bf41673feed8" containerName="revoke-keys" Jul 17 18:39:23.396067 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.396071 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="80e2218c-1cf2-47ea-9d81-039004147a3b" containerName="revoke-keys" Jul 17 18:39:23.396067 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.396076 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="80e2218c-1cf2-47ea-9d81-039004147a3b" containerName="revoke-keys" Jul 17 18:39:23.398890 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.398871 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn" Jul 17 18:39:23.401844 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.401814 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:39:23.401844 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.401829 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:39:23.406607 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.406581 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn"] Jul 17 18:39:23.579179 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.579143 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ec338b-556d-4ac3-a92d-3b56b382490f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-g45vn\" (UID: \"84ec338b-556d-4ac3-a92d-3b56b382490f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn" Jul 17 18:39:23.680375 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.680291 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ec338b-556d-4ac3-a92d-3b56b382490f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-g45vn\" (UID: \"84ec338b-556d-4ac3-a92d-3b56b382490f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn" Jul 17 18:39:23.680933 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.680910 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ec338b-556d-4ac3-a92d-3b56b382490f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-g45vn\" (UID: \"84ec338b-556d-4ac3-a92d-3b56b382490f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn" Jul 17 18:39:23.710350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.710327 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn" Jul 17 18:39:23.837848 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.837782 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn"] Jul 17 18:39:23.840187 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:39:23.840154 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ec338b_556d_4ac3_a92d_3b56b382490f.slice/crio-50dd377085eb4d6cc498ed7a6e49273b644d2d74d613253dcdd048c2c15ce7b9 WatchSource:0}: Error finding container 50dd377085eb4d6cc498ed7a6e49273b644d2d74d613253dcdd048c2c15ce7b9: Status 404 returned error can't find the container with id 50dd377085eb4d6cc498ed7a6e49273b644d2d74d613253dcdd048c2c15ce7b9 Jul 17 18:39:23.886275 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:23.886243 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn" event={"ID":"84ec338b-556d-4ac3-a92d-3b56b382490f","Type":"ContainerStarted","Data":"50dd377085eb4d6cc498ed7a6e49273b644d2d74d613253dcdd048c2c15ce7b9"} Jul 17 18:39:24.892329 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:24.892299 2572 generic.go:356] "Generic (PLEG): container finished" podID="84ec338b-556d-4ac3-a92d-3b56b382490f" containerID="1b1b10b62ba1ddbd2a784ec53868fbce741f1970f97d93c10e337140f9dd09a2" exitCode=6 Jul 17 18:39:24.892726 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:24.892383 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn" event={"ID":"84ec338b-556d-4ac3-a92d-3b56b382490f","Type":"ContainerDied","Data":"1b1b10b62ba1ddbd2a784ec53868fbce741f1970f97d93c10e337140f9dd09a2"} Jul 17 18:39:24.892726 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:24.892536 2572 scope.go:117] "RemoveContainer" containerID="1b1b10b62ba1ddbd2a784ec53868fbce741f1970f97d93c10e337140f9dd09a2" Jul 17 18:39:25.900002 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:25.899970 2572 generic.go:356] "Generic (PLEG): container finished" podID="84ec338b-556d-4ac3-a92d-3b56b382490f" containerID="8351f85f3e90dc287f564aad1bd8cd322041c12a2e3ae9f2a4cc71ba04c63ab8" exitCode=6 Jul 17 18:39:25.900412 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:25.900064 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn" event={"ID":"84ec338b-556d-4ac3-a92d-3b56b382490f","Type":"ContainerDied","Data":"8351f85f3e90dc287f564aad1bd8cd322041c12a2e3ae9f2a4cc71ba04c63ab8"} Jul 17 18:39:25.900412 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:25.900127 2572 scope.go:117] "RemoveContainer" containerID="1b1b10b62ba1ddbd2a784ec53868fbce741f1970f97d93c10e337140f9dd09a2" Jul 17 18:39:25.900412 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:25.900370 2572 scope.go:117] "RemoveContainer" containerID="8351f85f3e90dc287f564aad1bd8cd322041c12a2e3ae9f2a4cc71ba04c63ab8" Jul 17 18:39:25.900579 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:39:25.900558 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-g45vn_odh-ai-gateway-infra(84ec338b-556d-4ac3-a92d-3b56b382490f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn" podUID="84ec338b-556d-4ac3-a92d-3b56b382490f" Jul 17 18:39:26.905855 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:26.905827 2572 scope.go:117] "RemoveContainer" containerID="8351f85f3e90dc287f564aad1bd8cd322041c12a2e3ae9f2a4cc71ba04c63ab8" Jul 17 18:39:26.906256 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:39:26.906044 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-g45vn_odh-ai-gateway-infra(84ec338b-556d-4ac3-a92d-3b56b382490f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn" podUID="84ec338b-556d-4ac3-a92d-3b56b382490f" Jul 17 18:39:27.283147 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:27.283112 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc"] Jul 17 18:39:27.283470 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:27.283453 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b0ed0d-e2f5-485c-8339-bf41673feed8" containerName="revoke-keys" Jul 17 18:39:27.283554 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:27.283473 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b0ed0d-e2f5-485c-8339-bf41673feed8" containerName="revoke-keys" Jul 17 18:39:27.283607 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:27.283572 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b0ed0d-e2f5-485c-8339-bf41673feed8" containerName="revoke-keys" Jul 17 18:39:27.283607 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:27.283586 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="80e2218c-1cf2-47ea-9d81-039004147a3b" containerName="revoke-keys" Jul 17 18:39:27.287654 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:27.287633 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc" Jul 17 18:39:27.293430 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:27.293403 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc"] Jul 17 18:39:27.409402 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:27.409370 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2c303d2-a346-414e-b9f4-13bcadf85788-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc\" (UID: \"e2c303d2-a346-414e-b9f4-13bcadf85788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc" Jul 17 18:39:27.509944 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:27.509900 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2c303d2-a346-414e-b9f4-13bcadf85788-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc\" (UID: \"e2c303d2-a346-414e-b9f4-13bcadf85788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc" Jul 17 18:39:27.510561 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:27.510534 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2c303d2-a346-414e-b9f4-13bcadf85788-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc\" (UID: \"e2c303d2-a346-414e-b9f4-13bcadf85788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc" Jul 17 18:39:27.599148 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:27.599075 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc" Jul 17 18:39:27.724368 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:27.724342 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc"] Jul 17 18:39:27.726705 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:39:27.726663 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c303d2_a346_414e_b9f4_13bcadf85788.slice/crio-6ceb40a77921dc7f92a9557714854369c4f1a3f5a316f51f0ee58bf964e86732 WatchSource:0}: Error finding container 6ceb40a77921dc7f92a9557714854369c4f1a3f5a316f51f0ee58bf964e86732: Status 404 returned error can't find the container with id 6ceb40a77921dc7f92a9557714854369c4f1a3f5a316f51f0ee58bf964e86732 Jul 17 18:39:27.910403 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:27.910317 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc" event={"ID":"e2c303d2-a346-414e-b9f4-13bcadf85788","Type":"ContainerStarted","Data":"6ceb40a77921dc7f92a9557714854369c4f1a3f5a316f51f0ee58bf964e86732"} Jul 17 18:39:28.916255 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:28.916223 2572 generic.go:356] "Generic (PLEG): container finished" podID="e2c303d2-a346-414e-b9f4-13bcadf85788" containerID="75398757c87c36b8df6c1ad846d9cc9f7ba4608afa4bbb4afee2d815086e1679" exitCode=6 Jul 17 18:39:28.916692 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:28.916265 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc" event={"ID":"e2c303d2-a346-414e-b9f4-13bcadf85788","Type":"ContainerDied","Data":"75398757c87c36b8df6c1ad846d9cc9f7ba4608afa4bbb4afee2d815086e1679"} Jul 17 18:39:28.916692 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:28.916465 2572 scope.go:117] "RemoveContainer" containerID="75398757c87c36b8df6c1ad846d9cc9f7ba4608afa4bbb4afee2d815086e1679" Jul 17 18:39:29.922426 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:29.922340 2572 generic.go:356] "Generic (PLEG): container finished" podID="e2c303d2-a346-414e-b9f4-13bcadf85788" containerID="4e0d1c2694861d594d1f9430e92b9783ac71fc4d4c28758fcbc163b507365670" exitCode=6 Jul 17 18:39:29.922847 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:29.922422 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc" event={"ID":"e2c303d2-a346-414e-b9f4-13bcadf85788","Type":"ContainerDied","Data":"4e0d1c2694861d594d1f9430e92b9783ac71fc4d4c28758fcbc163b507365670"} Jul 17 18:39:29.922847 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:29.922479 2572 scope.go:117] "RemoveContainer" containerID="75398757c87c36b8df6c1ad846d9cc9f7ba4608afa4bbb4afee2d815086e1679" Jul 17 18:39:29.922847 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:29.922680 2572 scope.go:117] "RemoveContainer" containerID="4e0d1c2694861d594d1f9430e92b9783ac71fc4d4c28758fcbc163b507365670" Jul 17 18:39:29.922963 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:39:29.922876 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc_odh-ai-gateway-infra(e2c303d2-a346-414e-b9f4-13bcadf85788)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc" podUID="e2c303d2-a346-414e-b9f4-13bcadf85788" Jul 17 18:39:30.928028 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:30.927982 2572 scope.go:117] "RemoveContainer" containerID="4e0d1c2694861d594d1f9430e92b9783ac71fc4d4c28758fcbc163b507365670" Jul 17 18:39:30.928435 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:39:30.928172 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc_odh-ai-gateway-infra(e2c303d2-a346-414e-b9f4-13bcadf85788)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc" podUID="e2c303d2-a346-414e-b9f4-13bcadf85788" Jul 17 18:39:40.789462 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:40.789418 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt"] Jul 17 18:39:40.794206 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:40.794184 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt" Jul 17 18:39:40.800814 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:40.800787 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt"] Jul 17 18:39:40.811549 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:40.811520 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa46dd31-1e21-4b77-8bb2-944e2f007b91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt\" (UID: \"aa46dd31-1e21-4b77-8bb2-944e2f007b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt" Jul 17 18:39:40.911890 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:40.911864 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa46dd31-1e21-4b77-8bb2-944e2f007b91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt\" (UID: \"aa46dd31-1e21-4b77-8bb2-944e2f007b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt" Jul 17 18:39:40.912471 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:40.912453 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa46dd31-1e21-4b77-8bb2-944e2f007b91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt\" (UID: \"aa46dd31-1e21-4b77-8bb2-944e2f007b91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt" Jul 17 18:39:41.107056 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:41.106957 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt" Jul 17 18:39:41.219779 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:41.219754 2572 scope.go:117] "RemoveContainer" containerID="8351f85f3e90dc287f564aad1bd8cd322041c12a2e3ae9f2a4cc71ba04c63ab8" Jul 17 18:39:41.235114 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:41.235080 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt"] Jul 17 18:39:41.237922 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:39:41.237895 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa46dd31_1e21_4b77_8bb2_944e2f007b91.slice/crio-59de59b3df73e3d1d0e2a023a5cf8cd23f58afb0c2ceb05459438942fb4afb17 WatchSource:0}: Error finding container 59de59b3df73e3d1d0e2a023a5cf8cd23f58afb0c2ceb05459438942fb4afb17: Status 404 returned error can't find the container with id 59de59b3df73e3d1d0e2a023a5cf8cd23f58afb0c2ceb05459438942fb4afb17 Jul 17 18:39:41.976825 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:41.976795 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt" event={"ID":"aa46dd31-1e21-4b77-8bb2-944e2f007b91","Type":"ContainerStarted","Data":"7f697132a97e7943c5ae2da5deb83c75d3227724cdede47ffd43ce5a33f13cc8"} Jul 17 18:39:41.977262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:41.976875 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt" event={"ID":"aa46dd31-1e21-4b77-8bb2-944e2f007b91","Type":"ContainerStarted","Data":"59de59b3df73e3d1d0e2a023a5cf8cd23f58afb0c2ceb05459438942fb4afb17"} Jul 17 18:39:41.978618 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:41.978593 2572 generic.go:356] "Generic (PLEG): container finished" podID="84ec338b-556d-4ac3-a92d-3b56b382490f" containerID="bb678f2363782e30cb100777323ac0d15977793eaa41695a825f37daa4823af6" exitCode=6 Jul 17 18:39:41.978738 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:41.978662 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn" event={"ID":"84ec338b-556d-4ac3-a92d-3b56b382490f","Type":"ContainerDied","Data":"bb678f2363782e30cb100777323ac0d15977793eaa41695a825f37daa4823af6"} Jul 17 18:39:41.978738 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:41.978724 2572 scope.go:117] "RemoveContainer" containerID="8351f85f3e90dc287f564aad1bd8cd322041c12a2e3ae9f2a4cc71ba04c63ab8" Jul 17 18:39:41.978991 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:41.978973 2572 scope.go:117] "RemoveContainer" containerID="bb678f2363782e30cb100777323ac0d15977793eaa41695a825f37daa4823af6" Jul 17 18:39:41.979206 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:39:41.979184 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-g45vn_odh-ai-gateway-infra(84ec338b-556d-4ac3-a92d-3b56b382490f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn" podUID="84ec338b-556d-4ac3-a92d-3b56b382490f" Jul 17 18:39:41.991699 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:41.991663 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt" podStartSLOduration=1.991648919 podStartE2EDuration="1.991648919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:39:41.990865457 +0000 UTC m=+2577.400293062" watchObservedRunningTime="2026-07-17 18:39:41.991648919 +0000 UTC m=+2577.401076525" Jul 17 18:39:42.985031 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:42.984983 2572 generic.go:356] "Generic (PLEG): container finished" podID="aa46dd31-1e21-4b77-8bb2-944e2f007b91" containerID="7f697132a97e7943c5ae2da5deb83c75d3227724cdede47ffd43ce5a33f13cc8" exitCode=6 Jul 17 18:39:42.985473 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:42.985070 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt" event={"ID":"aa46dd31-1e21-4b77-8bb2-944e2f007b91","Type":"ContainerDied","Data":"7f697132a97e7943c5ae2da5deb83c75d3227724cdede47ffd43ce5a33f13cc8"} Jul 17 18:39:42.985473 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:42.985250 2572 scope.go:117] "RemoveContainer" containerID="7f697132a97e7943c5ae2da5deb83c75d3227724cdede47ffd43ce5a33f13cc8" Jul 17 18:39:43.014561 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:43.014536 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn"] Jul 17 18:39:43.136772 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:43.136750 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn" Jul 17 18:39:43.228408 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:43.228386 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ec338b-556d-4ac3-a92d-3b56b382490f-maas-api-service-ca\") pod \"84ec338b-556d-4ac3-a92d-3b56b382490f\" (UID: \"84ec338b-556d-4ac3-a92d-3b56b382490f\") " Jul 17 18:39:43.228723 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:43.228700 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84ec338b-556d-4ac3-a92d-3b56b382490f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "84ec338b-556d-4ac3-a92d-3b56b382490f" (UID: "84ec338b-556d-4ac3-a92d-3b56b382490f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:39:43.329671 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:43.329647 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84ec338b-556d-4ac3-a92d-3b56b382490f-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:39:43.992494 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:43.992406 2572 generic.go:356] "Generic (PLEG): container finished" podID="aa46dd31-1e21-4b77-8bb2-944e2f007b91" containerID="d2dc2edc05b9fe225eb025511c695a20ff764b01ad5c243041cb7e3530b2aa7a" exitCode=6 Jul 17 18:39:43.992925 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:43.992497 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt" event={"ID":"aa46dd31-1e21-4b77-8bb2-944e2f007b91","Type":"ContainerDied","Data":"d2dc2edc05b9fe225eb025511c695a20ff764b01ad5c243041cb7e3530b2aa7a"} Jul 17 18:39:43.992925 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:43.992578 2572 scope.go:117] "RemoveContainer" containerID="7f697132a97e7943c5ae2da5deb83c75d3227724cdede47ffd43ce5a33f13cc8" Jul 17 18:39:43.992925 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:43.992679 2572 scope.go:117] "RemoveContainer" containerID="d2dc2edc05b9fe225eb025511c695a20ff764b01ad5c243041cb7e3530b2aa7a" Jul 17 18:39:43.992925 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:39:43.992893 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt_odh-ai-gateway-infra(aa46dd31-1e21-4b77-8bb2-944e2f007b91)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt" podUID="aa46dd31-1e21-4b77-8bb2-944e2f007b91" Jul 17 18:39:43.993833 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:43.993816 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn" Jul 17 18:39:43.993910 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:43.993829 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn" event={"ID":"84ec338b-556d-4ac3-a92d-3b56b382490f","Type":"ContainerDied","Data":"50dd377085eb4d6cc498ed7a6e49273b644d2d74d613253dcdd048c2c15ce7b9"} Jul 17 18:39:44.003103 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:44.003085 2572 scope.go:117] "RemoveContainer" containerID="bb678f2363782e30cb100777323ac0d15977793eaa41695a825f37daa4823af6" Jul 17 18:39:44.052991 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:44.052969 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn"] Jul 17 18:39:44.058823 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:44.058803 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g45vn"] Jul 17 18:39:45.000217 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:45.000185 2572 scope.go:117] "RemoveContainer" containerID="d2dc2edc05b9fe225eb025511c695a20ff764b01ad5c243041cb7e3530b2aa7a" Jul 17 18:39:45.000627 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:39:45.000385 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt_odh-ai-gateway-infra(aa46dd31-1e21-4b77-8bb2-944e2f007b91)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt" podUID="aa46dd31-1e21-4b77-8bb2-944e2f007b91" Jul 17 18:39:45.221437 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:45.221408 2572 scope.go:117] "RemoveContainer" containerID="4e0d1c2694861d594d1f9430e92b9783ac71fc4d4c28758fcbc163b507365670" Jul 17 18:39:45.226496 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:45.226466 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84ec338b-556d-4ac3-a92d-3b56b382490f" path="/var/lib/kubelet/pods/84ec338b-556d-4ac3-a92d-3b56b382490f/volumes" Jul 17 18:39:46.007170 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:46.007136 2572 generic.go:356] "Generic (PLEG): container finished" podID="e2c303d2-a346-414e-b9f4-13bcadf85788" containerID="e0e57d6470f3a7fc1e3ef6355ea830e3667f945c8cd990d02181f28422c1dafe" exitCode=6 Jul 17 18:39:46.007550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:46.007202 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc" event={"ID":"e2c303d2-a346-414e-b9f4-13bcadf85788","Type":"ContainerDied","Data":"e0e57d6470f3a7fc1e3ef6355ea830e3667f945c8cd990d02181f28422c1dafe"} Jul 17 18:39:46.007550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:46.007263 2572 scope.go:117] "RemoveContainer" containerID="4e0d1c2694861d594d1f9430e92b9783ac71fc4d4c28758fcbc163b507365670" Jul 17 18:39:46.007550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:46.007536 2572 scope.go:117] "RemoveContainer" containerID="e0e57d6470f3a7fc1e3ef6355ea830e3667f945c8cd990d02181f28422c1dafe" Jul 17 18:39:46.007781 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:39:46.007752 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc_odh-ai-gateway-infra(e2c303d2-a346-414e-b9f4-13bcadf85788)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc" podUID="e2c303d2-a346-414e-b9f4-13bcadf85788" Jul 17 18:39:47.031782 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:47.031751 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc"] Jul 17 18:39:47.164740 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:47.164710 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc" Jul 17 18:39:47.257935 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:47.257910 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2c303d2-a346-414e-b9f4-13bcadf85788-maas-api-service-ca\") pod \"e2c303d2-a346-414e-b9f4-13bcadf85788\" (UID: \"e2c303d2-a346-414e-b9f4-13bcadf85788\") " Jul 17 18:39:47.258285 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:47.258263 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2c303d2-a346-414e-b9f4-13bcadf85788-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e2c303d2-a346-414e-b9f4-13bcadf85788" (UID: "e2c303d2-a346-414e-b9f4-13bcadf85788"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:39:47.359036 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:47.358936 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2c303d2-a346-414e-b9f4-13bcadf85788-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:39:48.018134 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:48.018106 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc" Jul 17 18:39:48.018282 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:48.018111 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc" event={"ID":"e2c303d2-a346-414e-b9f4-13bcadf85788","Type":"ContainerDied","Data":"6ceb40a77921dc7f92a9557714854369c4f1a3f5a316f51f0ee58bf964e86732"} Jul 17 18:39:48.018282 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:48.018243 2572 scope.go:117] "RemoveContainer" containerID="e0e57d6470f3a7fc1e3ef6355ea830e3667f945c8cd990d02181f28422c1dafe" Jul 17 18:39:48.043973 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:48.043947 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc"] Jul 17 18:39:48.048536 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:48.048515 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-rzwhc"] Jul 17 18:39:49.224472 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:49.224437 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2c303d2-a346-414e-b9f4-13bcadf85788" path="/var/lib/kubelet/pods/e2c303d2-a346-414e-b9f4-13bcadf85788/volumes" Jul 17 18:39:59.219552 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:39:59.219526 2572 scope.go:117] "RemoveContainer" containerID="d2dc2edc05b9fe225eb025511c695a20ff764b01ad5c243041cb7e3530b2aa7a" Jul 17 18:40:00.068078 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:00.068044 2572 generic.go:356] "Generic (PLEG): container finished" podID="aa46dd31-1e21-4b77-8bb2-944e2f007b91" containerID="02fb70f59cc3447b22d238e78ad524959dcb204fb6f42421100be447fb06fe88" exitCode=6 Jul 17 18:40:00.068259 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:00.068096 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt" event={"ID":"aa46dd31-1e21-4b77-8bb2-944e2f007b91","Type":"ContainerDied","Data":"02fb70f59cc3447b22d238e78ad524959dcb204fb6f42421100be447fb06fe88"} Jul 17 18:40:00.068259 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:00.068143 2572 scope.go:117] "RemoveContainer" containerID="d2dc2edc05b9fe225eb025511c695a20ff764b01ad5c243041cb7e3530b2aa7a" Jul 17 18:40:00.068448 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:00.068429 2572 scope.go:117] "RemoveContainer" containerID="02fb70f59cc3447b22d238e78ad524959dcb204fb6f42421100be447fb06fe88" Jul 17 18:40:00.068647 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:40:00.068625 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt_odh-ai-gateway-infra(aa46dd31-1e21-4b77-8bb2-944e2f007b91)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt" podUID="aa46dd31-1e21-4b77-8bb2-944e2f007b91" Jul 17 18:40:01.093622 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:01.093588 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt"] Jul 17 18:40:01.215663 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:01.215640 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt" Jul 17 18:40:01.274761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:01.274734 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa46dd31-1e21-4b77-8bb2-944e2f007b91-maas-api-service-ca\") pod \"aa46dd31-1e21-4b77-8bb2-944e2f007b91\" (UID: \"aa46dd31-1e21-4b77-8bb2-944e2f007b91\") " Jul 17 18:40:01.275073 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:01.275053 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa46dd31-1e21-4b77-8bb2-944e2f007b91-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aa46dd31-1e21-4b77-8bb2-944e2f007b91" (UID: "aa46dd31-1e21-4b77-8bb2-944e2f007b91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:40:01.375751 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:01.375683 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa46dd31-1e21-4b77-8bb2-944e2f007b91-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:40:02.079373 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:02.079341 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt" event={"ID":"aa46dd31-1e21-4b77-8bb2-944e2f007b91","Type":"ContainerDied","Data":"59de59b3df73e3d1d0e2a023a5cf8cd23f58afb0c2ceb05459438942fb4afb17"} Jul 17 18:40:02.079373 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:02.079369 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt" Jul 17 18:40:02.079600 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:02.079402 2572 scope.go:117] "RemoveContainer" containerID="02fb70f59cc3447b22d238e78ad524959dcb204fb6f42421100be447fb06fe88" Jul 17 18:40:02.098616 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:02.098590 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt"] Jul 17 18:40:02.109350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:02.109328 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-p26wt"] Jul 17 18:40:03.224687 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:03.224652 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa46dd31-1e21-4b77-8bb2-944e2f007b91" path="/var/lib/kubelet/pods/aa46dd31-1e21-4b77-8bb2-944e2f007b91/volumes" Jul 17 18:40:15.211602 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.211511 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr"] Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.211921 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2c303d2-a346-414e-b9f4-13bcadf85788" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.211941 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2c303d2-a346-414e-b9f4-13bcadf85788" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.211954 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2c303d2-a346-414e-b9f4-13bcadf85788" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.211961 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2c303d2-a346-414e-b9f4-13bcadf85788" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.211973 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa46dd31-1e21-4b77-8bb2-944e2f007b91" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.211980 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa46dd31-1e21-4b77-8bb2-944e2f007b91" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.211988 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa46dd31-1e21-4b77-8bb2-944e2f007b91" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.211995 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa46dd31-1e21-4b77-8bb2-944e2f007b91" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.212027 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2c303d2-a346-414e-b9f4-13bcadf85788" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.212035 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2c303d2-a346-414e-b9f4-13bcadf85788" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.212050 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84ec338b-556d-4ac3-a92d-3b56b382490f" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.212058 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="84ec338b-556d-4ac3-a92d-3b56b382490f" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.212080 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84ec338b-556d-4ac3-a92d-3b56b382490f" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.212088 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="84ec338b-556d-4ac3-a92d-3b56b382490f" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.212163 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="84ec338b-556d-4ac3-a92d-3b56b382490f" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.212173 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa46dd31-1e21-4b77-8bb2-944e2f007b91" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.212182 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa46dd31-1e21-4b77-8bb2-944e2f007b91" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.212193 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2c303d2-a346-414e-b9f4-13bcadf85788" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.212201 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2c303d2-a346-414e-b9f4-13bcadf85788" containerName="revoke-keys" Jul 17 18:40:15.212199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.212209 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="84ec338b-556d-4ac3-a92d-3b56b382490f" containerName="revoke-keys" Jul 17 18:40:15.213135 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.212217 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2c303d2-a346-414e-b9f4-13bcadf85788" containerName="revoke-keys" Jul 17 18:40:15.216888 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.216870 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr" Jul 17 18:40:15.219736 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.219715 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:40:15.219836 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.219773 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:40:15.225713 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.225687 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr"] Jul 17 18:40:15.296086 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.296057 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50dc1b88-3aa7-424f-ac15-b5d1677e4a5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr\" (UID: \"50dc1b88-3aa7-424f-ac15-b5d1677e4a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr" Jul 17 18:40:15.397335 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.397303 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50dc1b88-3aa7-424f-ac15-b5d1677e4a5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr\" (UID: \"50dc1b88-3aa7-424f-ac15-b5d1677e4a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr" Jul 17 18:40:15.397939 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.397919 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50dc1b88-3aa7-424f-ac15-b5d1677e4a5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr\" (UID: \"50dc1b88-3aa7-424f-ac15-b5d1677e4a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr" Jul 17 18:40:15.528429 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.528407 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr" Jul 17 18:40:15.859355 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:15.859333 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr"] Jul 17 18:40:15.861106 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:40:15.861074 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50dc1b88_3aa7_424f_ac15_b5d1677e4a5e.slice/crio-0e8bcfafb20f97b4e8447ee8e442b8b71039b1047100778e05e939cb9468bbe4 WatchSource:0}: Error finding container 0e8bcfafb20f97b4e8447ee8e442b8b71039b1047100778e05e939cb9468bbe4: Status 404 returned error can't find the container with id 0e8bcfafb20f97b4e8447ee8e442b8b71039b1047100778e05e939cb9468bbe4 Jul 17 18:40:16.136880 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:16.136815 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr" event={"ID":"50dc1b88-3aa7-424f-ac15-b5d1677e4a5e","Type":"ContainerStarted","Data":"9499b8c006a7ebd0704938fae01ec1e7fdae1810cfe19c7c2c8ec5c89057851e"} Jul 17 18:40:16.136880 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:16.136869 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr" event={"ID":"50dc1b88-3aa7-424f-ac15-b5d1677e4a5e","Type":"ContainerStarted","Data":"0e8bcfafb20f97b4e8447ee8e442b8b71039b1047100778e05e939cb9468bbe4"} Jul 17 18:40:16.158529 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:16.158482 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr" podStartSLOduration=1.158467364 podStartE2EDuration="1.158467364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:40:16.156643253 +0000 UTC m=+2611.566070859" watchObservedRunningTime="2026-07-17 18:40:16.158467364 +0000 UTC m=+2611.567895019" Jul 17 18:40:17.142271 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:17.142232 2572 generic.go:356] "Generic (PLEG): container finished" podID="50dc1b88-3aa7-424f-ac15-b5d1677e4a5e" containerID="9499b8c006a7ebd0704938fae01ec1e7fdae1810cfe19c7c2c8ec5c89057851e" exitCode=6 Jul 17 18:40:17.142666 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:17.142296 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr" event={"ID":"50dc1b88-3aa7-424f-ac15-b5d1677e4a5e","Type":"ContainerDied","Data":"9499b8c006a7ebd0704938fae01ec1e7fdae1810cfe19c7c2c8ec5c89057851e"} Jul 17 18:40:17.142666 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:17.142455 2572 scope.go:117] "RemoveContainer" containerID="9499b8c006a7ebd0704938fae01ec1e7fdae1810cfe19c7c2c8ec5c89057851e" Jul 17 18:40:18.147347 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:18.147313 2572 generic.go:356] "Generic (PLEG): container finished" podID="50dc1b88-3aa7-424f-ac15-b5d1677e4a5e" containerID="e2f6fb0ef6bb0aa35bd065a78b9a0c78a168decd9b280b37d54c31eb1449b384" exitCode=6 Jul 17 18:40:18.147730 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:18.147403 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr" event={"ID":"50dc1b88-3aa7-424f-ac15-b5d1677e4a5e","Type":"ContainerDied","Data":"e2f6fb0ef6bb0aa35bd065a78b9a0c78a168decd9b280b37d54c31eb1449b384"} Jul 17 18:40:18.147730 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:18.147464 2572 scope.go:117] "RemoveContainer" containerID="9499b8c006a7ebd0704938fae01ec1e7fdae1810cfe19c7c2c8ec5c89057851e" Jul 17 18:40:18.147730 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:18.147687 2572 scope.go:117] "RemoveContainer" containerID="e2f6fb0ef6bb0aa35bd065a78b9a0c78a168decd9b280b37d54c31eb1449b384" Jul 17 18:40:18.147882 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:40:18.147865 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr_odh-ai-gateway-infra(50dc1b88-3aa7-424f-ac15-b5d1677e4a5e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr" podUID="50dc1b88-3aa7-424f-ac15-b5d1677e4a5e" Jul 17 18:40:19.152813 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:19.152789 2572 scope.go:117] "RemoveContainer" containerID="e2f6fb0ef6bb0aa35bd065a78b9a0c78a168decd9b280b37d54c31eb1449b384" Jul 17 18:40:19.153185 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:40:19.152946 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr_odh-ai-gateway-infra(50dc1b88-3aa7-424f-ac15-b5d1677e4a5e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr" podUID="50dc1b88-3aa7-424f-ac15-b5d1677e4a5e" Jul 17 18:40:23.376328 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:23.376290 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp"] Jul 17 18:40:23.376684 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:23.376572 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa46dd31-1e21-4b77-8bb2-944e2f007b91" containerName="revoke-keys" Jul 17 18:40:23.376684 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:23.376582 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa46dd31-1e21-4b77-8bb2-944e2f007b91" containerName="revoke-keys" Jul 17 18:40:23.376684 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:23.376603 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84ec338b-556d-4ac3-a92d-3b56b382490f" containerName="revoke-keys" Jul 17 18:40:23.376684 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:23.376608 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="84ec338b-556d-4ac3-a92d-3b56b382490f" containerName="revoke-keys" Jul 17 18:40:23.376684 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:23.376656 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa46dd31-1e21-4b77-8bb2-944e2f007b91" containerName="revoke-keys" Jul 17 18:40:23.376684 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:23.376668 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="84ec338b-556d-4ac3-a92d-3b56b382490f" containerName="revoke-keys" Jul 17 18:40:23.380979 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:23.380949 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp" Jul 17 18:40:23.387231 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:23.387202 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp"] Jul 17 18:40:23.468600 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:23.468566 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c8a282-8c2f-4edb-bbe9-57f373df6646-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp\" (UID: \"71c8a282-8c2f-4edb-bbe9-57f373df6646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp" Jul 17 18:40:23.569198 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:23.569154 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c8a282-8c2f-4edb-bbe9-57f373df6646-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp\" (UID: \"71c8a282-8c2f-4edb-bbe9-57f373df6646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp" Jul 17 18:40:23.569759 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:23.569738 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c8a282-8c2f-4edb-bbe9-57f373df6646-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp\" (UID: \"71c8a282-8c2f-4edb-bbe9-57f373df6646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp" Jul 17 18:40:23.693144 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:23.693073 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp" Jul 17 18:40:24.025080 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:24.025052 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp"] Jul 17 18:40:24.026993 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:40:24.026965 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c8a282_8c2f_4edb_bbe9_57f373df6646.slice/crio-9bf90e20801ce6a7281bd724c3fce066afefeab32618db5aed9ef7794f945f3e WatchSource:0}: Error finding container 9bf90e20801ce6a7281bd724c3fce066afefeab32618db5aed9ef7794f945f3e: Status 404 returned error can't find the container with id 9bf90e20801ce6a7281bd724c3fce066afefeab32618db5aed9ef7794f945f3e Jul 17 18:40:24.174328 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:24.174297 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp" event={"ID":"71c8a282-8c2f-4edb-bbe9-57f373df6646","Type":"ContainerStarted","Data":"9bf90e20801ce6a7281bd724c3fce066afefeab32618db5aed9ef7794f945f3e"} Jul 17 18:40:25.182289 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:25.182249 2572 generic.go:356] "Generic (PLEG): container finished" podID="71c8a282-8c2f-4edb-bbe9-57f373df6646" containerID="84e601535a54036d60a9495c377190d25aabbe7d6b0c1d4ad89796fd10209371" exitCode=6 Jul 17 18:40:25.182743 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:25.182334 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp" event={"ID":"71c8a282-8c2f-4edb-bbe9-57f373df6646","Type":"ContainerDied","Data":"84e601535a54036d60a9495c377190d25aabbe7d6b0c1d4ad89796fd10209371"} Jul 17 18:40:25.182743 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:25.182594 2572 scope.go:117] "RemoveContainer" containerID="84e601535a54036d60a9495c377190d25aabbe7d6b0c1d4ad89796fd10209371" Jul 17 18:40:26.188294 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:26.188258 2572 generic.go:356] "Generic (PLEG): container finished" podID="71c8a282-8c2f-4edb-bbe9-57f373df6646" containerID="1a18e527f3f3f73210920568d284a8aa13d0d829057582d2dbdc9129b0ee63e9" exitCode=6 Jul 17 18:40:26.188731 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:26.188345 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp" event={"ID":"71c8a282-8c2f-4edb-bbe9-57f373df6646","Type":"ContainerDied","Data":"1a18e527f3f3f73210920568d284a8aa13d0d829057582d2dbdc9129b0ee63e9"} Jul 17 18:40:26.188731 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:26.188393 2572 scope.go:117] "RemoveContainer" containerID="84e601535a54036d60a9495c377190d25aabbe7d6b0c1d4ad89796fd10209371" Jul 17 18:40:26.188731 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:26.188590 2572 scope.go:117] "RemoveContainer" containerID="1a18e527f3f3f73210920568d284a8aa13d0d829057582d2dbdc9129b0ee63e9" Jul 17 18:40:26.188867 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:40:26.188842 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp_odh-ai-gateway-infra(71c8a282-8c2f-4edb-bbe9-57f373df6646)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp" podUID="71c8a282-8c2f-4edb-bbe9-57f373df6646" Jul 17 18:40:27.194292 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:27.194264 2572 scope.go:117] "RemoveContainer" containerID="1a18e527f3f3f73210920568d284a8aa13d0d829057582d2dbdc9129b0ee63e9" Jul 17 18:40:27.194682 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:40:27.194454 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp_odh-ai-gateway-infra(71c8a282-8c2f-4edb-bbe9-57f373df6646)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp" podUID="71c8a282-8c2f-4edb-bbe9-57f373df6646" Jul 17 18:40:30.219888 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:30.219859 2572 scope.go:117] "RemoveContainer" containerID="e2f6fb0ef6bb0aa35bd065a78b9a0c78a168decd9b280b37d54c31eb1449b384" Jul 17 18:40:31.212254 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:31.212170 2572 generic.go:356] "Generic (PLEG): container finished" podID="50dc1b88-3aa7-424f-ac15-b5d1677e4a5e" containerID="6380bf10529a2803c666e36ff73995a5be7f1408506d1a740e57db8624a1b287" exitCode=6 Jul 17 18:40:31.212254 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:31.212242 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr" event={"ID":"50dc1b88-3aa7-424f-ac15-b5d1677e4a5e","Type":"ContainerDied","Data":"6380bf10529a2803c666e36ff73995a5be7f1408506d1a740e57db8624a1b287"} Jul 17 18:40:31.212479 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:31.212304 2572 scope.go:117] "RemoveContainer" containerID="e2f6fb0ef6bb0aa35bd065a78b9a0c78a168decd9b280b37d54c31eb1449b384" Jul 17 18:40:31.212604 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:31.212589 2572 scope.go:117] "RemoveContainer" containerID="6380bf10529a2803c666e36ff73995a5be7f1408506d1a740e57db8624a1b287" Jul 17 18:40:31.212822 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:40:31.212798 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr_odh-ai-gateway-infra(50dc1b88-3aa7-424f-ac15-b5d1677e4a5e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr" podUID="50dc1b88-3aa7-424f-ac15-b5d1677e4a5e" Jul 17 18:40:32.236592 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:32.236559 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr"] Jul 17 18:40:32.359597 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:32.359576 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr" Jul 17 18:40:32.547751 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:32.547725 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50dc1b88-3aa7-424f-ac15-b5d1677e4a5e-maas-api-service-ca\") pod \"50dc1b88-3aa7-424f-ac15-b5d1677e4a5e\" (UID: \"50dc1b88-3aa7-424f-ac15-b5d1677e4a5e\") " Jul 17 18:40:32.548052 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:32.548023 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50dc1b88-3aa7-424f-ac15-b5d1677e4a5e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50dc1b88-3aa7-424f-ac15-b5d1677e4a5e" (UID: "50dc1b88-3aa7-424f-ac15-b5d1677e4a5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:40:32.648883 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:32.648855 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50dc1b88-3aa7-424f-ac15-b5d1677e4a5e-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:40:33.223488 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:33.223460 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr" Jul 17 18:40:33.224137 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:33.224115 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr" event={"ID":"50dc1b88-3aa7-424f-ac15-b5d1677e4a5e","Type":"ContainerDied","Data":"0e8bcfafb20f97b4e8447ee8e442b8b71039b1047100778e05e939cb9468bbe4"} Jul 17 18:40:33.224200 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:33.224163 2572 scope.go:117] "RemoveContainer" containerID="6380bf10529a2803c666e36ff73995a5be7f1408506d1a740e57db8624a1b287" Jul 17 18:40:33.249433 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:33.249402 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr"] Jul 17 18:40:33.254747 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:33.254723 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qpqzr"] Jul 17 18:40:35.224427 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:35.224383 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50dc1b88-3aa7-424f-ac15-b5d1677e4a5e" path="/var/lib/kubelet/pods/50dc1b88-3aa7-424f-ac15-b5d1677e4a5e/volumes" Jul 17 18:40:40.834837 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:40.834805 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv"] Jul 17 18:40:40.835348 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:40.835280 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50dc1b88-3aa7-424f-ac15-b5d1677e4a5e" containerName="revoke-keys" Jul 17 18:40:40.835348 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:40.835298 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="50dc1b88-3aa7-424f-ac15-b5d1677e4a5e" containerName="revoke-keys" Jul 17 18:40:40.835348 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:40.835313 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50dc1b88-3aa7-424f-ac15-b5d1677e4a5e" containerName="revoke-keys" Jul 17 18:40:40.835348 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:40.835322 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="50dc1b88-3aa7-424f-ac15-b5d1677e4a5e" containerName="revoke-keys" Jul 17 18:40:40.835348 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:40.835336 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50dc1b88-3aa7-424f-ac15-b5d1677e4a5e" containerName="revoke-keys" Jul 17 18:40:40.835348 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:40.835344 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="50dc1b88-3aa7-424f-ac15-b5d1677e4a5e" containerName="revoke-keys" Jul 17 18:40:40.835647 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:40.835429 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="50dc1b88-3aa7-424f-ac15-b5d1677e4a5e" containerName="revoke-keys" Jul 17 18:40:40.835647 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:40.835441 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="50dc1b88-3aa7-424f-ac15-b5d1677e4a5e" containerName="revoke-keys" Jul 17 18:40:40.839614 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:40.839593 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv" Jul 17 18:40:40.846180 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:40.846154 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv"] Jul 17 18:40:40.909289 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:40.909258 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f9a556-1318-4dec-aff1-2e3d8d330727-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv\" (UID: \"b5f9a556-1318-4dec-aff1-2e3d8d330727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv" Jul 17 18:40:41.009876 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:41.009848 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f9a556-1318-4dec-aff1-2e3d8d330727-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv\" (UID: \"b5f9a556-1318-4dec-aff1-2e3d8d330727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv" Jul 17 18:40:41.010460 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:41.010441 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f9a556-1318-4dec-aff1-2e3d8d330727-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv\" (UID: \"b5f9a556-1318-4dec-aff1-2e3d8d330727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv" Jul 17 18:40:41.151156 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:41.151072 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv" Jul 17 18:40:41.277426 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:41.277399 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv"] Jul 17 18:40:42.220111 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:42.219989 2572 scope.go:117] "RemoveContainer" containerID="1a18e527f3f3f73210920568d284a8aa13d0d829057582d2dbdc9129b0ee63e9" Jul 17 18:40:42.267459 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:42.267434 2572 generic.go:356] "Generic (PLEG): container finished" podID="b5f9a556-1318-4dec-aff1-2e3d8d330727" containerID="397891b975501f16d1a4d63ed9fd5db6b1e58613e4276861e69e0f6252d7bbbe" exitCode=6 Jul 17 18:40:42.267559 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:42.267527 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv" event={"ID":"b5f9a556-1318-4dec-aff1-2e3d8d330727","Type":"ContainerDied","Data":"397891b975501f16d1a4d63ed9fd5db6b1e58613e4276861e69e0f6252d7bbbe"} Jul 17 18:40:42.267615 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:42.267587 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv" event={"ID":"b5f9a556-1318-4dec-aff1-2e3d8d330727","Type":"ContainerStarted","Data":"d7f5965ae0d5eb18b908a8bad149a1e989a4c691675006dcb67ec20578c3d486"} Jul 17 18:40:42.267699 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:42.267688 2572 scope.go:117] "RemoveContainer" containerID="397891b975501f16d1a4d63ed9fd5db6b1e58613e4276861e69e0f6252d7bbbe" Jul 17 18:40:43.273273 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:43.273240 2572 generic.go:356] "Generic (PLEG): container finished" podID="71c8a282-8c2f-4edb-bbe9-57f373df6646" containerID="b033593485ff3d26e9e86e394f84dafeb3b6dccd06e01a337d485aca40f2cc6c" exitCode=6 Jul 17 18:40:43.273757 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:43.273314 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp" event={"ID":"71c8a282-8c2f-4edb-bbe9-57f373df6646","Type":"ContainerDied","Data":"b033593485ff3d26e9e86e394f84dafeb3b6dccd06e01a337d485aca40f2cc6c"} Jul 17 18:40:43.273757 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:43.273378 2572 scope.go:117] "RemoveContainer" containerID="1a18e527f3f3f73210920568d284a8aa13d0d829057582d2dbdc9129b0ee63e9" Jul 17 18:40:43.273862 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:43.273781 2572 scope.go:117] "RemoveContainer" containerID="b033593485ff3d26e9e86e394f84dafeb3b6dccd06e01a337d485aca40f2cc6c" Jul 17 18:40:43.274029 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:40:43.273989 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp_odh-ai-gateway-infra(71c8a282-8c2f-4edb-bbe9-57f373df6646)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp" podUID="71c8a282-8c2f-4edb-bbe9-57f373df6646" Jul 17 18:40:43.275257 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:43.275239 2572 generic.go:356] "Generic (PLEG): container finished" podID="b5f9a556-1318-4dec-aff1-2e3d8d330727" containerID="6cc5d9dc782a77dfda4d660f3fa89492d510f558b49116ed2ffd480630152c24" exitCode=6 Jul 17 18:40:43.275345 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:43.275262 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv" event={"ID":"b5f9a556-1318-4dec-aff1-2e3d8d330727","Type":"ContainerDied","Data":"6cc5d9dc782a77dfda4d660f3fa89492d510f558b49116ed2ffd480630152c24"} Jul 17 18:40:43.275549 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:43.275536 2572 scope.go:117] "RemoveContainer" containerID="6cc5d9dc782a77dfda4d660f3fa89492d510f558b49116ed2ffd480630152c24" Jul 17 18:40:43.275702 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:40:43.275686 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv_odh-ai-gateway-infra(b5f9a556-1318-4dec-aff1-2e3d8d330727)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv" podUID="b5f9a556-1318-4dec-aff1-2e3d8d330727" Jul 17 18:40:43.283991 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:43.283974 2572 scope.go:117] "RemoveContainer" containerID="397891b975501f16d1a4d63ed9fd5db6b1e58613e4276861e69e0f6252d7bbbe" Jul 17 18:40:44.282520 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:44.282491 2572 scope.go:117] "RemoveContainer" containerID="6cc5d9dc782a77dfda4d660f3fa89492d510f558b49116ed2ffd480630152c24" Jul 17 18:40:44.282917 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:40:44.282675 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv_odh-ai-gateway-infra(b5f9a556-1318-4dec-aff1-2e3d8d330727)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv" podUID="b5f9a556-1318-4dec-aff1-2e3d8d330727" Jul 17 18:40:44.303138 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:44.303105 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp"] Jul 17 18:40:44.421232 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:44.421207 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp" Jul 17 18:40:44.434584 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:44.434558 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c8a282-8c2f-4edb-bbe9-57f373df6646-maas-api-service-ca\") pod \"71c8a282-8c2f-4edb-bbe9-57f373df6646\" (UID: \"71c8a282-8c2f-4edb-bbe9-57f373df6646\") " Jul 17 18:40:44.434918 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:44.434897 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c8a282-8c2f-4edb-bbe9-57f373df6646-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "71c8a282-8c2f-4edb-bbe9-57f373df6646" (UID: "71c8a282-8c2f-4edb-bbe9-57f373df6646"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:40:44.535248 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:44.535170 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c8a282-8c2f-4edb-bbe9-57f373df6646-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:40:45.287558 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:45.287524 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp" event={"ID":"71c8a282-8c2f-4edb-bbe9-57f373df6646","Type":"ContainerDied","Data":"9bf90e20801ce6a7281bd724c3fce066afefeab32618db5aed9ef7794f945f3e"} Jul 17 18:40:45.287967 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:45.287590 2572 scope.go:117] "RemoveContainer" containerID="b033593485ff3d26e9e86e394f84dafeb3b6dccd06e01a337d485aca40f2cc6c" Jul 17 18:40:45.287967 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:45.287540 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp" Jul 17 18:40:45.307774 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:45.307746 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp"] Jul 17 18:40:45.317316 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:45.317284 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vrdbp"] Jul 17 18:40:47.224519 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:47.224487 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71c8a282-8c2f-4edb-bbe9-57f373df6646" path="/var/lib/kubelet/pods/71c8a282-8c2f-4edb-bbe9-57f373df6646/volumes" Jul 17 18:40:59.219714 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:40:59.219679 2572 scope.go:117] "RemoveContainer" containerID="6cc5d9dc782a77dfda4d660f3fa89492d510f558b49116ed2ffd480630152c24" Jul 17 18:41:00.352169 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:00.352134 2572 generic.go:356] "Generic (PLEG): container finished" podID="b5f9a556-1318-4dec-aff1-2e3d8d330727" containerID="7063904cd43a566b3f23e968de68c6ab3a953447a6597d5c9a0a6ed92ab6ae38" exitCode=6 Jul 17 18:41:00.352629 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:00.352185 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv" event={"ID":"b5f9a556-1318-4dec-aff1-2e3d8d330727","Type":"ContainerDied","Data":"7063904cd43a566b3f23e968de68c6ab3a953447a6597d5c9a0a6ed92ab6ae38"} Jul 17 18:41:00.352629 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:00.352276 2572 scope.go:117] "RemoveContainer" containerID="6cc5d9dc782a77dfda4d660f3fa89492d510f558b49116ed2ffd480630152c24" Jul 17 18:41:00.352629 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:00.352616 2572 scope.go:117] "RemoveContainer" containerID="7063904cd43a566b3f23e968de68c6ab3a953447a6597d5c9a0a6ed92ab6ae38" Jul 17 18:41:00.352817 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:41:00.352793 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv_odh-ai-gateway-infra(b5f9a556-1318-4dec-aff1-2e3d8d330727)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv" podUID="b5f9a556-1318-4dec-aff1-2e3d8d330727" Jul 17 18:41:01.377392 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:01.377360 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv"] Jul 17 18:41:01.503167 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:01.503142 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv" Jul 17 18:41:01.566000 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:01.565972 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f9a556-1318-4dec-aff1-2e3d8d330727-maas-api-service-ca\") pod \"b5f9a556-1318-4dec-aff1-2e3d8d330727\" (UID: \"b5f9a556-1318-4dec-aff1-2e3d8d330727\") " Jul 17 18:41:01.566295 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:01.566273 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f9a556-1318-4dec-aff1-2e3d8d330727-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b5f9a556-1318-4dec-aff1-2e3d8d330727" (UID: "b5f9a556-1318-4dec-aff1-2e3d8d330727"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:41:01.666506 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:01.666442 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f9a556-1318-4dec-aff1-2e3d8d330727-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:41:02.361965 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:02.361926 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv" event={"ID":"b5f9a556-1318-4dec-aff1-2e3d8d330727","Type":"ContainerDied","Data":"d7f5965ae0d5eb18b908a8bad149a1e989a4c691675006dcb67ec20578c3d486"} Jul 17 18:41:02.361965 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:02.361965 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv" Jul 17 18:41:02.362214 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:02.361997 2572 scope.go:117] "RemoveContainer" containerID="7063904cd43a566b3f23e968de68c6ab3a953447a6597d5c9a0a6ed92ab6ae38" Jul 17 18:41:02.381701 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:02.381671 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv"] Jul 17 18:41:02.387781 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:02.387761 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-xnrcv"] Jul 17 18:41:03.224376 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:03.224341 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5f9a556-1318-4dec-aff1-2e3d8d330727" path="/var/lib/kubelet/pods/b5f9a556-1318-4dec-aff1-2e3d8d330727/volumes" Jul 17 18:41:23.421689 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.421652 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56"] Jul 17 18:41:23.422158 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.421943 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f9a556-1318-4dec-aff1-2e3d8d330727" containerName="revoke-keys" Jul 17 18:41:23.422158 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.421953 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f9a556-1318-4dec-aff1-2e3d8d330727" containerName="revoke-keys" Jul 17 18:41:23.422158 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.421966 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c8a282-8c2f-4edb-bbe9-57f373df6646" containerName="revoke-keys" Jul 17 18:41:23.422158 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.421972 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c8a282-8c2f-4edb-bbe9-57f373df6646" containerName="revoke-keys" Jul 17 18:41:23.422158 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.421980 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c8a282-8c2f-4edb-bbe9-57f373df6646" containerName="revoke-keys" Jul 17 18:41:23.422158 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.421986 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c8a282-8c2f-4edb-bbe9-57f373df6646" containerName="revoke-keys" Jul 17 18:41:23.422158 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.422000 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f9a556-1318-4dec-aff1-2e3d8d330727" containerName="revoke-keys" Jul 17 18:41:23.422158 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.422020 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f9a556-1318-4dec-aff1-2e3d8d330727" containerName="revoke-keys" Jul 17 18:41:23.422158 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.422086 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="71c8a282-8c2f-4edb-bbe9-57f373df6646" containerName="revoke-keys" Jul 17 18:41:23.422158 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.422094 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="50dc1b88-3aa7-424f-ac15-b5d1677e4a5e" containerName="revoke-keys" Jul 17 18:41:23.422158 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.422102 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f9a556-1318-4dec-aff1-2e3d8d330727" containerName="revoke-keys" Jul 17 18:41:23.422158 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.422109 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="71c8a282-8c2f-4edb-bbe9-57f373df6646" containerName="revoke-keys" Jul 17 18:41:23.422158 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.422115 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f9a556-1318-4dec-aff1-2e3d8d330727" containerName="revoke-keys" Jul 17 18:41:23.422158 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.422121 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f9a556-1318-4dec-aff1-2e3d8d330727" containerName="revoke-keys" Jul 17 18:41:23.424784 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.424768 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56" Jul 17 18:41:23.427241 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.427219 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:41:23.427455 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.427442 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:41:23.433429 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.433404 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56"] Jul 17 18:41:23.552982 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.552945 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffae54aa-7c53-4bd4-a0b9-91d922ac87f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-kss56\" (UID: \"ffae54aa-7c53-4bd4-a0b9-91d922ac87f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56" Jul 17 18:41:23.654230 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.654194 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffae54aa-7c53-4bd4-a0b9-91d922ac87f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-kss56\" (UID: \"ffae54aa-7c53-4bd4-a0b9-91d922ac87f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56" Jul 17 18:41:23.654819 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.654800 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffae54aa-7c53-4bd4-a0b9-91d922ac87f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-kss56\" (UID: \"ffae54aa-7c53-4bd4-a0b9-91d922ac87f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56" Jul 17 18:41:23.736848 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.736768 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56" Jul 17 18:41:23.860199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:23.860172 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56"] Jul 17 18:41:23.861929 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:41:23.861901 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffae54aa_7c53_4bd4_a0b9_91d922ac87f8.slice/crio-a76f18bc96e59a0f6b5628e3dc3f87f9466259415448ad831f0a30ad44a704df WatchSource:0}: Error finding container a76f18bc96e59a0f6b5628e3dc3f87f9466259415448ad831f0a30ad44a704df: Status 404 returned error can't find the container with id a76f18bc96e59a0f6b5628e3dc3f87f9466259415448ad831f0a30ad44a704df Jul 17 18:41:24.454528 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:24.454498 2572 generic.go:356] "Generic (PLEG): container finished" podID="ffae54aa-7c53-4bd4-a0b9-91d922ac87f8" containerID="2f259e0c5431d881e0f14e7df66a0aa53a2e91173b9fc9364548035bad04bbc3" exitCode=6 Jul 17 18:41:24.454881 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:24.454579 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56" event={"ID":"ffae54aa-7c53-4bd4-a0b9-91d922ac87f8","Type":"ContainerDied","Data":"2f259e0c5431d881e0f14e7df66a0aa53a2e91173b9fc9364548035bad04bbc3"} Jul 17 18:41:24.454881 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:24.454632 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56" event={"ID":"ffae54aa-7c53-4bd4-a0b9-91d922ac87f8","Type":"ContainerStarted","Data":"a76f18bc96e59a0f6b5628e3dc3f87f9466259415448ad831f0a30ad44a704df"} Jul 17 18:41:24.454881 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:24.454823 2572 scope.go:117] "RemoveContainer" containerID="2f259e0c5431d881e0f14e7df66a0aa53a2e91173b9fc9364548035bad04bbc3" Jul 17 18:41:25.460457 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:25.460368 2572 generic.go:356] "Generic (PLEG): container finished" podID="ffae54aa-7c53-4bd4-a0b9-91d922ac87f8" containerID="ffea9da0cf179de6b98c077a62437370753ff9f25eb29a7cd4cd2061d4dad026" exitCode=6 Jul 17 18:41:25.460874 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:25.460447 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56" event={"ID":"ffae54aa-7c53-4bd4-a0b9-91d922ac87f8","Type":"ContainerDied","Data":"ffea9da0cf179de6b98c077a62437370753ff9f25eb29a7cd4cd2061d4dad026"} Jul 17 18:41:25.460874 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:25.460515 2572 scope.go:117] "RemoveContainer" containerID="2f259e0c5431d881e0f14e7df66a0aa53a2e91173b9fc9364548035bad04bbc3" Jul 17 18:41:25.460874 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:25.460652 2572 scope.go:117] "RemoveContainer" containerID="ffea9da0cf179de6b98c077a62437370753ff9f25eb29a7cd4cd2061d4dad026" Jul 17 18:41:25.460874 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:41:25.460835 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-kss56_odh-ai-gateway-infra(ffae54aa-7c53-4bd4-a0b9-91d922ac87f8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56" podUID="ffae54aa-7c53-4bd4-a0b9-91d922ac87f8" Jul 17 18:41:26.466526 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:26.466493 2572 scope.go:117] "RemoveContainer" containerID="ffea9da0cf179de6b98c077a62437370753ff9f25eb29a7cd4cd2061d4dad026" Jul 17 18:41:26.466914 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:41:26.466670 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-kss56_odh-ai-gateway-infra(ffae54aa-7c53-4bd4-a0b9-91d922ac87f8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56" podUID="ffae54aa-7c53-4bd4-a0b9-91d922ac87f8" Jul 17 18:41:41.220178 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:41.220093 2572 scope.go:117] "RemoveContainer" containerID="ffea9da0cf179de6b98c077a62437370753ff9f25eb29a7cd4cd2061d4dad026" Jul 17 18:41:42.533919 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:42.533887 2572 generic.go:356] "Generic (PLEG): container finished" podID="ffae54aa-7c53-4bd4-a0b9-91d922ac87f8" containerID="08a0e9499a1eab92053105b4377b8c1c7adeff81e24622096aa73debe66955d1" exitCode=6 Jul 17 18:41:42.534354 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:42.533961 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56" event={"ID":"ffae54aa-7c53-4bd4-a0b9-91d922ac87f8","Type":"ContainerDied","Data":"08a0e9499a1eab92053105b4377b8c1c7adeff81e24622096aa73debe66955d1"} Jul 17 18:41:42.534354 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:42.534043 2572 scope.go:117] "RemoveContainer" containerID="ffea9da0cf179de6b98c077a62437370753ff9f25eb29a7cd4cd2061d4dad026" Jul 17 18:41:42.534354 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:42.534306 2572 scope.go:117] "RemoveContainer" containerID="08a0e9499a1eab92053105b4377b8c1c7adeff81e24622096aa73debe66955d1" Jul 17 18:41:42.534549 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:41:42.534527 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-kss56_odh-ai-gateway-infra(ffae54aa-7c53-4bd4-a0b9-91d922ac87f8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56" podUID="ffae54aa-7c53-4bd4-a0b9-91d922ac87f8" Jul 17 18:41:43.559713 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:43.559682 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56"] Jul 17 18:41:43.684176 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:43.684145 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56" Jul 17 18:41:43.720917 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:43.720890 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffae54aa-7c53-4bd4-a0b9-91d922ac87f8-maas-api-service-ca\") pod \"ffae54aa-7c53-4bd4-a0b9-91d922ac87f8\" (UID: \"ffae54aa-7c53-4bd4-a0b9-91d922ac87f8\") " Jul 17 18:41:43.721251 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:43.721226 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffae54aa-7c53-4bd4-a0b9-91d922ac87f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ffae54aa-7c53-4bd4-a0b9-91d922ac87f8" (UID: "ffae54aa-7c53-4bd4-a0b9-91d922ac87f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:41:43.821476 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:43.821406 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffae54aa-7c53-4bd4-a0b9-91d922ac87f8-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:41:44.544530 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:44.544497 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56" Jul 17 18:41:44.544530 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:44.544519 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56" event={"ID":"ffae54aa-7c53-4bd4-a0b9-91d922ac87f8","Type":"ContainerDied","Data":"a76f18bc96e59a0f6b5628e3dc3f87f9466259415448ad831f0a30ad44a704df"} Jul 17 18:41:44.544780 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:44.544579 2572 scope.go:117] "RemoveContainer" containerID="08a0e9499a1eab92053105b4377b8c1c7adeff81e24622096aa73debe66955d1" Jul 17 18:41:44.569747 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:44.568229 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56"] Jul 17 18:41:44.572637 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:44.572614 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-kss56"] Jul 17 18:41:45.224231 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:41:45.224201 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffae54aa-7c53-4bd4-a0b9-91d922ac87f8" path="/var/lib/kubelet/pods/ffae54aa-7c53-4bd4-a0b9-91d922ac87f8/volumes" Jul 17 18:42:05.300953 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.300917 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g"] Jul 17 18:42:05.301359 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.301213 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c8a282-8c2f-4edb-bbe9-57f373df6646" containerName="revoke-keys" Jul 17 18:42:05.301359 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.301225 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c8a282-8c2f-4edb-bbe9-57f373df6646" containerName="revoke-keys" Jul 17 18:42:05.301359 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.301234 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffae54aa-7c53-4bd4-a0b9-91d922ac87f8" containerName="revoke-keys" Jul 17 18:42:05.301359 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.301240 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffae54aa-7c53-4bd4-a0b9-91d922ac87f8" containerName="revoke-keys" Jul 17 18:42:05.301359 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.301249 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffae54aa-7c53-4bd4-a0b9-91d922ac87f8" containerName="revoke-keys" Jul 17 18:42:05.301359 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.301254 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffae54aa-7c53-4bd4-a0b9-91d922ac87f8" containerName="revoke-keys" Jul 17 18:42:05.301359 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.301267 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f9a556-1318-4dec-aff1-2e3d8d330727" containerName="revoke-keys" Jul 17 18:42:05.301359 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.301273 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f9a556-1318-4dec-aff1-2e3d8d330727" containerName="revoke-keys" Jul 17 18:42:05.301359 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.301324 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="71c8a282-8c2f-4edb-bbe9-57f373df6646" containerName="revoke-keys" Jul 17 18:42:05.301359 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.301331 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffae54aa-7c53-4bd4-a0b9-91d922ac87f8" containerName="revoke-keys" Jul 17 18:42:05.301359 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.301340 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffae54aa-7c53-4bd4-a0b9-91d922ac87f8" containerName="revoke-keys" Jul 17 18:42:05.303951 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.303936 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g" Jul 17 18:42:05.306323 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.306305 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:42:05.306410 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.306326 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:42:05.311353 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.311331 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g"] Jul 17 18:42:05.403508 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.403475 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7043de-74ac-43f6-9f64-ab60a6bfa6ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g\" (UID: \"3d7043de-74ac-43f6-9f64-ab60a6bfa6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g" Jul 17 18:42:05.504365 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.504332 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7043de-74ac-43f6-9f64-ab60a6bfa6ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g\" (UID: \"3d7043de-74ac-43f6-9f64-ab60a6bfa6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g" Jul 17 18:42:05.504916 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.504896 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7043de-74ac-43f6-9f64-ab60a6bfa6ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g\" (UID: \"3d7043de-74ac-43f6-9f64-ab60a6bfa6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g" Jul 17 18:42:05.615412 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.615335 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g" Jul 17 18:42:05.737035 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:05.736999 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g"] Jul 17 18:42:05.739115 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:42:05.739088 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7043de_74ac_43f6_9f64_ab60a6bfa6ba.slice/crio-9a46acc5e3ed65aacc8150843277a6a3eba9a913307619f79aa3040b622f4cac WatchSource:0}: Error finding container 9a46acc5e3ed65aacc8150843277a6a3eba9a913307619f79aa3040b622f4cac: Status 404 returned error can't find the container with id 9a46acc5e3ed65aacc8150843277a6a3eba9a913307619f79aa3040b622f4cac Jul 17 18:42:06.639047 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:06.638942 2572 generic.go:356] "Generic (PLEG): container finished" podID="3d7043de-74ac-43f6-9f64-ab60a6bfa6ba" containerID="d5f4432e7c9d58cd530347f7879cc3ade6fb8acc090b1684c9c38cd712893026" exitCode=6 Jul 17 18:42:06.639047 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:06.638990 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g" event={"ID":"3d7043de-74ac-43f6-9f64-ab60a6bfa6ba","Type":"ContainerDied","Data":"d5f4432e7c9d58cd530347f7879cc3ade6fb8acc090b1684c9c38cd712893026"} Jul 17 18:42:06.639047 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:06.639046 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g" event={"ID":"3d7043de-74ac-43f6-9f64-ab60a6bfa6ba","Type":"ContainerStarted","Data":"9a46acc5e3ed65aacc8150843277a6a3eba9a913307619f79aa3040b622f4cac"} Jul 17 18:42:06.639509 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:06.639218 2572 scope.go:117] "RemoveContainer" containerID="d5f4432e7c9d58cd530347f7879cc3ade6fb8acc090b1684c9c38cd712893026" Jul 17 18:42:07.645727 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:07.645691 2572 generic.go:356] "Generic (PLEG): container finished" podID="3d7043de-74ac-43f6-9f64-ab60a6bfa6ba" containerID="fb01ba5f7b1aac4c03ff6e15a3836a274f3871b2849e523da8bc5d0e1efbc416" exitCode=6 Jul 17 18:42:07.646230 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:07.645733 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g" event={"ID":"3d7043de-74ac-43f6-9f64-ab60a6bfa6ba","Type":"ContainerDied","Data":"fb01ba5f7b1aac4c03ff6e15a3836a274f3871b2849e523da8bc5d0e1efbc416"} Jul 17 18:42:07.646230 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:07.645794 2572 scope.go:117] "RemoveContainer" containerID="d5f4432e7c9d58cd530347f7879cc3ade6fb8acc090b1684c9c38cd712893026" Jul 17 18:42:07.646230 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:07.645971 2572 scope.go:117] "RemoveContainer" containerID="fb01ba5f7b1aac4c03ff6e15a3836a274f3871b2849e523da8bc5d0e1efbc416" Jul 17 18:42:07.646230 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:42:07.646216 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g_odh-ai-gateway-infra(3d7043de-74ac-43f6-9f64-ab60a6bfa6ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g" podUID="3d7043de-74ac-43f6-9f64-ab60a6bfa6ba" Jul 17 18:42:08.650895 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:08.650869 2572 scope.go:117] "RemoveContainer" containerID="fb01ba5f7b1aac4c03ff6e15a3836a274f3871b2849e523da8bc5d0e1efbc416" Jul 17 18:42:08.651324 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:42:08.651084 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g_odh-ai-gateway-infra(3d7043de-74ac-43f6-9f64-ab60a6bfa6ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g" podUID="3d7043de-74ac-43f6-9f64-ab60a6bfa6ba" Jul 17 18:42:17.420521 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:17.420485 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw"] Jul 17 18:42:17.421804 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:17.421774 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffae54aa-7c53-4bd4-a0b9-91d922ac87f8" containerName="revoke-keys" Jul 17 18:42:17.421970 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:17.421958 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffae54aa-7c53-4bd4-a0b9-91d922ac87f8" containerName="revoke-keys" Jul 17 18:42:17.422243 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:17.422227 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffae54aa-7c53-4bd4-a0b9-91d922ac87f8" containerName="revoke-keys" Jul 17 18:42:17.427455 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:17.427427 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw" Jul 17 18:42:17.429432 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:17.429406 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw"] Jul 17 18:42:17.506487 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:17.506451 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/027da4ab-64eb-483b-ad13-737da15629f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw\" (UID: \"027da4ab-64eb-483b-ad13-737da15629f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw" Jul 17 18:42:17.606942 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:17.606913 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/027da4ab-64eb-483b-ad13-737da15629f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw\" (UID: \"027da4ab-64eb-483b-ad13-737da15629f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw" Jul 17 18:42:17.607583 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:17.607563 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/027da4ab-64eb-483b-ad13-737da15629f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw\" (UID: \"027da4ab-64eb-483b-ad13-737da15629f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw" Jul 17 18:42:17.740329 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:17.740258 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw" Jul 17 18:42:17.863679 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:17.863657 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw"] Jul 17 18:42:17.866062 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:42:17.866033 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027da4ab_64eb_483b_ad13_737da15629f5.slice/crio-f610bd7de64f372d5712edf9dc62a409a42e3249099ee2ec2bcafe0a566273e7 WatchSource:0}: Error finding container f610bd7de64f372d5712edf9dc62a409a42e3249099ee2ec2bcafe0a566273e7: Status 404 returned error can't find the container with id f610bd7de64f372d5712edf9dc62a409a42e3249099ee2ec2bcafe0a566273e7 Jul 17 18:42:18.693985 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:18.693896 2572 generic.go:356] "Generic (PLEG): container finished" podID="027da4ab-64eb-483b-ad13-737da15629f5" containerID="1204fe89d57b539627b7f884c09f7ee854125c82aec8e00368ba19d70b2a3505" exitCode=6 Jul 17 18:42:18.694379 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:18.693989 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw" event={"ID":"027da4ab-64eb-483b-ad13-737da15629f5","Type":"ContainerDied","Data":"1204fe89d57b539627b7f884c09f7ee854125c82aec8e00368ba19d70b2a3505"} Jul 17 18:42:18.694379 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:18.694058 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw" event={"ID":"027da4ab-64eb-483b-ad13-737da15629f5","Type":"ContainerStarted","Data":"f610bd7de64f372d5712edf9dc62a409a42e3249099ee2ec2bcafe0a566273e7"} Jul 17 18:42:18.694379 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:18.694135 2572 scope.go:117] "RemoveContainer" containerID="1204fe89d57b539627b7f884c09f7ee854125c82aec8e00368ba19d70b2a3505" Jul 17 18:42:19.699804 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:19.699766 2572 generic.go:356] "Generic (PLEG): container finished" podID="027da4ab-64eb-483b-ad13-737da15629f5" containerID="a2242cfbbd5f619e6669a8b1f9f3948ab6b02c6aae44e78dc1b883c65839eede" exitCode=6 Jul 17 18:42:19.699804 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:19.699804 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw" event={"ID":"027da4ab-64eb-483b-ad13-737da15629f5","Type":"ContainerDied","Data":"a2242cfbbd5f619e6669a8b1f9f3948ab6b02c6aae44e78dc1b883c65839eede"} Jul 17 18:42:19.700371 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:19.699863 2572 scope.go:117] "RemoveContainer" containerID="1204fe89d57b539627b7f884c09f7ee854125c82aec8e00368ba19d70b2a3505" Jul 17 18:42:19.700371 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:19.700045 2572 scope.go:117] "RemoveContainer" containerID="a2242cfbbd5f619e6669a8b1f9f3948ab6b02c6aae44e78dc1b883c65839eede" Jul 17 18:42:19.700371 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:42:19.700264 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw_odh-ai-gateway-infra(027da4ab-64eb-483b-ad13-737da15629f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw" podUID="027da4ab-64eb-483b-ad13-737da15629f5" Jul 17 18:42:20.219512 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:20.219482 2572 scope.go:117] "RemoveContainer" containerID="fb01ba5f7b1aac4c03ff6e15a3836a274f3871b2849e523da8bc5d0e1efbc416" Jul 17 18:42:20.706055 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:20.705948 2572 scope.go:117] "RemoveContainer" containerID="a2242cfbbd5f619e6669a8b1f9f3948ab6b02c6aae44e78dc1b883c65839eede" Jul 17 18:42:20.706464 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:42:20.706163 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw_odh-ai-gateway-infra(027da4ab-64eb-483b-ad13-737da15629f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw" podUID="027da4ab-64eb-483b-ad13-737da15629f5" Jul 17 18:42:20.707382 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:20.707359 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g" event={"ID":"3d7043de-74ac-43f6-9f64-ab60a6bfa6ba","Type":"ContainerStarted","Data":"e54d2342650cae458dbd95b041297bbbab1e219eec0a23e510c58522085a904c"} Jul 17 18:42:20.737100 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:20.737054 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g" podStartSLOduration=15.737040553 podStartE2EDuration="15.737040553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:42:20.73536123 +0000 UTC m=+2736.144788837" watchObservedRunningTime="2026-07-17 18:42:20.737040553 +0000 UTC m=+2736.146468158" Jul 17 18:42:21.712845 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:21.712810 2572 generic.go:356] "Generic (PLEG): container finished" podID="3d7043de-74ac-43f6-9f64-ab60a6bfa6ba" containerID="e54d2342650cae458dbd95b041297bbbab1e219eec0a23e510c58522085a904c" exitCode=6 Jul 17 18:42:21.713318 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:21.712882 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g" event={"ID":"3d7043de-74ac-43f6-9f64-ab60a6bfa6ba","Type":"ContainerDied","Data":"e54d2342650cae458dbd95b041297bbbab1e219eec0a23e510c58522085a904c"} Jul 17 18:42:21.713318 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:21.712944 2572 scope.go:117] "RemoveContainer" containerID="fb01ba5f7b1aac4c03ff6e15a3836a274f3871b2849e523da8bc5d0e1efbc416" Jul 17 18:42:21.713318 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:21.713252 2572 scope.go:117] "RemoveContainer" containerID="e54d2342650cae458dbd95b041297bbbab1e219eec0a23e510c58522085a904c" Jul 17 18:42:21.713449 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:42:21.713430 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g_odh-ai-gateway-infra(3d7043de-74ac-43f6-9f64-ab60a6bfa6ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g" podUID="3d7043de-74ac-43f6-9f64-ab60a6bfa6ba" Jul 17 18:42:21.745269 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:21.745247 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g"] Jul 17 18:42:22.850622 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:22.850600 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g" Jul 17 18:42:22.953910 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:22.953881 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7043de-74ac-43f6-9f64-ab60a6bfa6ba-maas-api-service-ca\") pod \"3d7043de-74ac-43f6-9f64-ab60a6bfa6ba\" (UID: \"3d7043de-74ac-43f6-9f64-ab60a6bfa6ba\") " Jul 17 18:42:22.954225 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:22.954203 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7043de-74ac-43f6-9f64-ab60a6bfa6ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d7043de-74ac-43f6-9f64-ab60a6bfa6ba" (UID: "3d7043de-74ac-43f6-9f64-ab60a6bfa6ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:42:23.054802 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.054776 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7043de-74ac-43f6-9f64-ab60a6bfa6ba-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:42:23.467815 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.467734 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn"] Jul 17 18:42:23.468046 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.468034 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7043de-74ac-43f6-9f64-ab60a6bfa6ba" containerName="revoke-keys" Jul 17 18:42:23.468095 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.468048 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7043de-74ac-43f6-9f64-ab60a6bfa6ba" containerName="revoke-keys" Jul 17 18:42:23.468095 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.468063 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7043de-74ac-43f6-9f64-ab60a6bfa6ba" containerName="revoke-keys" Jul 17 18:42:23.468095 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.468068 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7043de-74ac-43f6-9f64-ab60a6bfa6ba" containerName="revoke-keys" Jul 17 18:42:23.468095 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.468075 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7043de-74ac-43f6-9f64-ab60a6bfa6ba" containerName="revoke-keys" Jul 17 18:42:23.468095 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.468081 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7043de-74ac-43f6-9f64-ab60a6bfa6ba" containerName="revoke-keys" Jul 17 18:42:23.468251 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.468134 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d7043de-74ac-43f6-9f64-ab60a6bfa6ba" containerName="revoke-keys" Jul 17 18:42:23.468251 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.468141 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d7043de-74ac-43f6-9f64-ab60a6bfa6ba" containerName="revoke-keys" Jul 17 18:42:23.470922 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.470905 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn" Jul 17 18:42:23.478039 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.477998 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn"] Jul 17 18:42:23.560269 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.560235 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0875077b-264b-41c3-9495-606a3c4736ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn\" (UID: \"0875077b-264b-41c3-9495-606a3c4736ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn" Jul 17 18:42:23.661440 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.661410 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0875077b-264b-41c3-9495-606a3c4736ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn\" (UID: \"0875077b-264b-41c3-9495-606a3c4736ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn" Jul 17 18:42:23.662058 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.662039 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0875077b-264b-41c3-9495-606a3c4736ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn\" (UID: \"0875077b-264b-41c3-9495-606a3c4736ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn" Jul 17 18:42:23.723400 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.723325 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g" event={"ID":"3d7043de-74ac-43f6-9f64-ab60a6bfa6ba","Type":"ContainerDied","Data":"9a46acc5e3ed65aacc8150843277a6a3eba9a913307619f79aa3040b622f4cac"} Jul 17 18:42:23.723400 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.723385 2572 scope.go:117] "RemoveContainer" containerID="e54d2342650cae458dbd95b041297bbbab1e219eec0a23e510c58522085a904c" Jul 17 18:42:23.723553 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.723337 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g" Jul 17 18:42:23.739994 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.739972 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g"] Jul 17 18:42:23.743765 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.743742 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5929g"] Jul 17 18:42:23.782153 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.782132 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn" Jul 17 18:42:23.922873 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:23.922851 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn"] Jul 17 18:42:23.924585 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:42:23.924556 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0875077b_264b_41c3_9495_606a3c4736ce.slice/crio-4ab9174999e0966e5848d1f2a457b4a145199163bff915a654bc23105d6fb410 WatchSource:0}: Error finding container 4ab9174999e0966e5848d1f2a457b4a145199163bff915a654bc23105d6fb410: Status 404 returned error can't find the container with id 4ab9174999e0966e5848d1f2a457b4a145199163bff915a654bc23105d6fb410 Jul 17 18:42:24.729553 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:24.729518 2572 generic.go:356] "Generic (PLEG): container finished" podID="0875077b-264b-41c3-9495-606a3c4736ce" containerID="15b0a6553bd729427e8fa25a9e70bb9639c84c645e1ec1afa5e6cb7c95969cd4" exitCode=6 Jul 17 18:42:24.729731 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:24.729611 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn" event={"ID":"0875077b-264b-41c3-9495-606a3c4736ce","Type":"ContainerDied","Data":"15b0a6553bd729427e8fa25a9e70bb9639c84c645e1ec1afa5e6cb7c95969cd4"} Jul 17 18:42:24.729731 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:24.729661 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn" event={"ID":"0875077b-264b-41c3-9495-606a3c4736ce","Type":"ContainerStarted","Data":"4ab9174999e0966e5848d1f2a457b4a145199163bff915a654bc23105d6fb410"} Jul 17 18:42:24.729809 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:24.729779 2572 scope.go:117] "RemoveContainer" containerID="15b0a6553bd729427e8fa25a9e70bb9639c84c645e1ec1afa5e6cb7c95969cd4" Jul 17 18:42:25.224668 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:25.224593 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d7043de-74ac-43f6-9f64-ab60a6bfa6ba" path="/var/lib/kubelet/pods/3d7043de-74ac-43f6-9f64-ab60a6bfa6ba/volumes" Jul 17 18:42:25.735864 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:25.735828 2572 generic.go:356] "Generic (PLEG): container finished" podID="0875077b-264b-41c3-9495-606a3c4736ce" containerID="e083d0415a4a9a6327874e5fe9a354d3a95cae7168bad9a7da2d197ec21e4784" exitCode=6 Jul 17 18:42:25.736072 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:25.735908 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn" event={"ID":"0875077b-264b-41c3-9495-606a3c4736ce","Type":"ContainerDied","Data":"e083d0415a4a9a6327874e5fe9a354d3a95cae7168bad9a7da2d197ec21e4784"} Jul 17 18:42:25.736072 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:25.735972 2572 scope.go:117] "RemoveContainer" containerID="15b0a6553bd729427e8fa25a9e70bb9639c84c645e1ec1afa5e6cb7c95969cd4" Jul 17 18:42:25.736213 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:25.736199 2572 scope.go:117] "RemoveContainer" containerID="e083d0415a4a9a6327874e5fe9a354d3a95cae7168bad9a7da2d197ec21e4784" Jul 17 18:42:25.736448 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:42:25.736430 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn_odh-ai-gateway-infra(0875077b-264b-41c3-9495-606a3c4736ce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn" podUID="0875077b-264b-41c3-9495-606a3c4736ce" Jul 17 18:42:26.742030 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:26.741988 2572 scope.go:117] "RemoveContainer" containerID="e083d0415a4a9a6327874e5fe9a354d3a95cae7168bad9a7da2d197ec21e4784" Jul 17 18:42:26.742378 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:42:26.742198 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn_odh-ai-gateway-infra(0875077b-264b-41c3-9495-606a3c4736ce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn" podUID="0875077b-264b-41c3-9495-606a3c4736ce" Jul 17 18:42:30.927051 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:30.927002 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq"] Jul 17 18:42:30.927428 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:30.927373 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d7043de-74ac-43f6-9f64-ab60a6bfa6ba" containerName="revoke-keys" Jul 17 18:42:30.931535 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:30.931518 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq" Jul 17 18:42:30.939000 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:30.938977 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq"] Jul 17 18:42:31.025654 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:31.025618 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e81bcc-8565-4faf-bee4-623239646e71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq\" (UID: \"a0e81bcc-8565-4faf-bee4-623239646e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq" Jul 17 18:42:31.126630 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:31.126603 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e81bcc-8565-4faf-bee4-623239646e71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq\" (UID: \"a0e81bcc-8565-4faf-bee4-623239646e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq" Jul 17 18:42:31.127210 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:31.127194 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e81bcc-8565-4faf-bee4-623239646e71-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq\" (UID: \"a0e81bcc-8565-4faf-bee4-623239646e71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq" Jul 17 18:42:31.242401 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:31.242330 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq" Jul 17 18:42:31.367057 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:31.367031 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq"] Jul 17 18:42:31.370046 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:42:31.369973 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e81bcc_8565_4faf_bee4_623239646e71.slice/crio-a66aa0278c22f1f32a9e6643d8e5be34c38ba9b22c15a0725ace9a2f0a0a9e49 WatchSource:0}: Error finding container a66aa0278c22f1f32a9e6643d8e5be34c38ba9b22c15a0725ace9a2f0a0a9e49: Status 404 returned error can't find the container with id a66aa0278c22f1f32a9e6643d8e5be34c38ba9b22c15a0725ace9a2f0a0a9e49 Jul 17 18:42:31.763456 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:31.763417 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq" event={"ID":"a0e81bcc-8565-4faf-bee4-623239646e71","Type":"ContainerStarted","Data":"46522755c75053d99190c433e2be97b888d78d927613216f382511bddc8b1eb6"} Jul 17 18:42:31.763609 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:31.763468 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq" event={"ID":"a0e81bcc-8565-4faf-bee4-623239646e71","Type":"ContainerStarted","Data":"a66aa0278c22f1f32a9e6643d8e5be34c38ba9b22c15a0725ace9a2f0a0a9e49"} Jul 17 18:42:31.779072 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:31.778964 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq" podStartSLOduration=1.778948918 podStartE2EDuration="1.778948918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:42:31.778179671 +0000 UTC m=+2747.187607287" watchObservedRunningTime="2026-07-17 18:42:31.778948918 +0000 UTC m=+2747.188376576" Jul 17 18:42:32.769204 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:32.769165 2572 generic.go:356] "Generic (PLEG): container finished" podID="a0e81bcc-8565-4faf-bee4-623239646e71" containerID="46522755c75053d99190c433e2be97b888d78d927613216f382511bddc8b1eb6" exitCode=6 Jul 17 18:42:32.769585 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:32.769246 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq" event={"ID":"a0e81bcc-8565-4faf-bee4-623239646e71","Type":"ContainerDied","Data":"46522755c75053d99190c433e2be97b888d78d927613216f382511bddc8b1eb6"} Jul 17 18:42:32.769585 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:32.769444 2572 scope.go:117] "RemoveContainer" containerID="46522755c75053d99190c433e2be97b888d78d927613216f382511bddc8b1eb6" Jul 17 18:42:33.220176 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:33.220145 2572 scope.go:117] "RemoveContainer" containerID="a2242cfbbd5f619e6669a8b1f9f3948ab6b02c6aae44e78dc1b883c65839eede" Jul 17 18:42:33.775322 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:33.775243 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw" event={"ID":"027da4ab-64eb-483b-ad13-737da15629f5","Type":"ContainerStarted","Data":"4924beafa4b6516397b45928663a6d64863dd700936079d60efe5c9cb74d6658"} Jul 17 18:42:33.776818 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:33.776788 2572 generic.go:356] "Generic (PLEG): container finished" podID="a0e81bcc-8565-4faf-bee4-623239646e71" containerID="9b754b625bac5988478e1607acbfc30e63d9567fb3b4b4287e464362e8345d4b" exitCode=6 Jul 17 18:42:33.776952 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:33.776825 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq" event={"ID":"a0e81bcc-8565-4faf-bee4-623239646e71","Type":"ContainerDied","Data":"9b754b625bac5988478e1607acbfc30e63d9567fb3b4b4287e464362e8345d4b"} Jul 17 18:42:33.776952 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:33.776888 2572 scope.go:117] "RemoveContainer" containerID="46522755c75053d99190c433e2be97b888d78d927613216f382511bddc8b1eb6" Jul 17 18:42:33.777051 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:33.777035 2572 scope.go:117] "RemoveContainer" containerID="9b754b625bac5988478e1607acbfc30e63d9567fb3b4b4287e464362e8345d4b" Jul 17 18:42:33.777230 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:42:33.777207 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq_odh-ai-gateway-infra(a0e81bcc-8565-4faf-bee4-623239646e71)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq" podUID="a0e81bcc-8565-4faf-bee4-623239646e71" Jul 17 18:42:33.791141 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:33.791103 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw" podStartSLOduration=16.791090567 podStartE2EDuration="16.791090567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:42:33.789850827 +0000 UTC m=+2749.199278436" watchObservedRunningTime="2026-07-17 18:42:33.791090567 +0000 UTC m=+2749.200518170" Jul 17 18:42:34.782991 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:34.782960 2572 scope.go:117] "RemoveContainer" containerID="9b754b625bac5988478e1607acbfc30e63d9567fb3b4b4287e464362e8345d4b" Jul 17 18:42:34.783419 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:42:34.783214 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq_odh-ai-gateway-infra(a0e81bcc-8565-4faf-bee4-623239646e71)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq" podUID="a0e81bcc-8565-4faf-bee4-623239646e71" Jul 17 18:42:34.784503 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:34.784479 2572 generic.go:356] "Generic (PLEG): container finished" podID="027da4ab-64eb-483b-ad13-737da15629f5" containerID="4924beafa4b6516397b45928663a6d64863dd700936079d60efe5c9cb74d6658" exitCode=6 Jul 17 18:42:34.784619 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:34.784512 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw" event={"ID":"027da4ab-64eb-483b-ad13-737da15629f5","Type":"ContainerDied","Data":"4924beafa4b6516397b45928663a6d64863dd700936079d60efe5c9cb74d6658"} Jul 17 18:42:34.784619 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:34.784566 2572 scope.go:117] "RemoveContainer" containerID="a2242cfbbd5f619e6669a8b1f9f3948ab6b02c6aae44e78dc1b883c65839eede" Jul 17 18:42:34.784783 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:34.784766 2572 scope.go:117] "RemoveContainer" containerID="4924beafa4b6516397b45928663a6d64863dd700936079d60efe5c9cb74d6658" Jul 17 18:42:34.784955 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:42:34.784934 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw_odh-ai-gateway-infra(027da4ab-64eb-483b-ad13-737da15629f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw" podUID="027da4ab-64eb-483b-ad13-737da15629f5" Jul 17 18:42:34.801681 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:34.801662 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw"] Jul 17 18:42:35.921595 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:35.921572 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw" Jul 17 18:42:35.969047 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:35.968994 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/027da4ab-64eb-483b-ad13-737da15629f5-maas-api-service-ca\") pod \"027da4ab-64eb-483b-ad13-737da15629f5\" (UID: \"027da4ab-64eb-483b-ad13-737da15629f5\") " Jul 17 18:42:35.969297 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:35.969275 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027da4ab-64eb-483b-ad13-737da15629f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "027da4ab-64eb-483b-ad13-737da15629f5" (UID: "027da4ab-64eb-483b-ad13-737da15629f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:42:36.070268 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:36.070201 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/027da4ab-64eb-483b-ad13-737da15629f5-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:42:36.795949 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:36.795917 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw" Jul 17 18:42:36.796147 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:36.795917 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw" event={"ID":"027da4ab-64eb-483b-ad13-737da15629f5","Type":"ContainerDied","Data":"f610bd7de64f372d5712edf9dc62a409a42e3249099ee2ec2bcafe0a566273e7"} Jul 17 18:42:36.796147 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:36.796058 2572 scope.go:117] "RemoveContainer" containerID="4924beafa4b6516397b45928663a6d64863dd700936079d60efe5c9cb74d6658" Jul 17 18:42:36.816269 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:36.816247 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw"] Jul 17 18:42:36.821977 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:36.821956 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lhhxw"] Jul 17 18:42:37.228414 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:37.228332 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="027da4ab-64eb-483b-ad13-737da15629f5" path="/var/lib/kubelet/pods/027da4ab-64eb-483b-ad13-737da15629f5/volumes" Jul 17 18:42:38.220296 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:38.220265 2572 scope.go:117] "RemoveContainer" containerID="e083d0415a4a9a6327874e5fe9a354d3a95cae7168bad9a7da2d197ec21e4784" Jul 17 18:42:38.807050 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:38.806960 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn" event={"ID":"0875077b-264b-41c3-9495-606a3c4736ce","Type":"ContainerStarted","Data":"9e456eab072cfc3c5fbbc384aa702c761e04510c9e9ae8d171959766b37a75f2"} Jul 17 18:42:38.823691 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:38.823642 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn" podStartSLOduration=15.823627306 podStartE2EDuration="15.823627306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:42:38.822215781 +0000 UTC m=+2754.231643388" watchObservedRunningTime="2026-07-17 18:42:38.823627306 +0000 UTC m=+2754.233054911" Jul 17 18:42:39.815917 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:39.815883 2572 generic.go:356] "Generic (PLEG): container finished" podID="0875077b-264b-41c3-9495-606a3c4736ce" containerID="9e456eab072cfc3c5fbbc384aa702c761e04510c9e9ae8d171959766b37a75f2" exitCode=6 Jul 17 18:42:39.816383 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:39.815960 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn" event={"ID":"0875077b-264b-41c3-9495-606a3c4736ce","Type":"ContainerDied","Data":"9e456eab072cfc3c5fbbc384aa702c761e04510c9e9ae8d171959766b37a75f2"} Jul 17 18:42:39.816383 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:39.816052 2572 scope.go:117] "RemoveContainer" containerID="e083d0415a4a9a6327874e5fe9a354d3a95cae7168bad9a7da2d197ec21e4784" Jul 17 18:42:39.816383 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:39.816344 2572 scope.go:117] "RemoveContainer" containerID="9e456eab072cfc3c5fbbc384aa702c761e04510c9e9ae8d171959766b37a75f2" Jul 17 18:42:39.816557 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:42:39.816542 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn_odh-ai-gateway-infra(0875077b-264b-41c3-9495-606a3c4736ce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn" podUID="0875077b-264b-41c3-9495-606a3c4736ce" Jul 17 18:42:39.832372 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:39.832348 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn"] Jul 17 18:42:40.940615 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:40.940590 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn" Jul 17 18:42:41.011328 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:41.011299 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0875077b-264b-41c3-9495-606a3c4736ce-maas-api-service-ca\") pod \"0875077b-264b-41c3-9495-606a3c4736ce\" (UID: \"0875077b-264b-41c3-9495-606a3c4736ce\") " Jul 17 18:42:41.011627 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:41.011604 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0875077b-264b-41c3-9495-606a3c4736ce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0875077b-264b-41c3-9495-606a3c4736ce" (UID: "0875077b-264b-41c3-9495-606a3c4736ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:42:41.112425 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:41.112346 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0875077b-264b-41c3-9495-606a3c4736ce-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:42:41.828021 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:41.827994 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn" Jul 17 18:42:41.828199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:41.827988 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn" event={"ID":"0875077b-264b-41c3-9495-606a3c4736ce","Type":"ContainerDied","Data":"4ab9174999e0966e5848d1f2a457b4a145199163bff915a654bc23105d6fb410"} Jul 17 18:42:41.828199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:41.828145 2572 scope.go:117] "RemoveContainer" containerID="9e456eab072cfc3c5fbbc384aa702c761e04510c9e9ae8d171959766b37a75f2" Jul 17 18:42:41.845367 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:41.845342 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn"] Jul 17 18:42:41.850958 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:41.850937 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-2wrcn"] Jul 17 18:42:43.224251 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:43.224217 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0875077b-264b-41c3-9495-606a3c4736ce" path="/var/lib/kubelet/pods/0875077b-264b-41c3-9495-606a3c4736ce/volumes" Jul 17 18:42:45.221179 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:45.221156 2572 scope.go:117] "RemoveContainer" containerID="9b754b625bac5988478e1607acbfc30e63d9567fb3b4b4287e464362e8345d4b" Jul 17 18:42:45.846336 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:45.846308 2572 generic.go:356] "Generic (PLEG): container finished" podID="a0e81bcc-8565-4faf-bee4-623239646e71" containerID="44d4630d4ad58868d65474f2f15dac4af4a9801fdbc97984561b4218214713af" exitCode=6 Jul 17 18:42:45.846509 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:45.846349 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq" event={"ID":"a0e81bcc-8565-4faf-bee4-623239646e71","Type":"ContainerDied","Data":"44d4630d4ad58868d65474f2f15dac4af4a9801fdbc97984561b4218214713af"} Jul 17 18:42:45.846509 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:45.846424 2572 scope.go:117] "RemoveContainer" containerID="9b754b625bac5988478e1607acbfc30e63d9567fb3b4b4287e464362e8345d4b" Jul 17 18:42:45.846718 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:45.846695 2572 scope.go:117] "RemoveContainer" containerID="44d4630d4ad58868d65474f2f15dac4af4a9801fdbc97984561b4218214713af" Jul 17 18:42:45.846951 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:42:45.846928 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq_odh-ai-gateway-infra(a0e81bcc-8565-4faf-bee4-623239646e71)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq" podUID="a0e81bcc-8565-4faf-bee4-623239646e71" Jul 17 18:42:46.870895 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:46.870864 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq"] Jul 17 18:42:46.996993 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:46.996969 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq" Jul 17 18:42:47.064283 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:47.064253 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e81bcc-8565-4faf-bee4-623239646e71-maas-api-service-ca\") pod \"a0e81bcc-8565-4faf-bee4-623239646e71\" (UID: \"a0e81bcc-8565-4faf-bee4-623239646e71\") " Jul 17 18:42:47.064610 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:47.064591 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e81bcc-8565-4faf-bee4-623239646e71-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0e81bcc-8565-4faf-bee4-623239646e71" (UID: "a0e81bcc-8565-4faf-bee4-623239646e71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:42:47.165425 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:47.165354 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0e81bcc-8565-4faf-bee4-623239646e71-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:42:47.856827 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:47.856791 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq" event={"ID":"a0e81bcc-8565-4faf-bee4-623239646e71","Type":"ContainerDied","Data":"a66aa0278c22f1f32a9e6643d8e5be34c38ba9b22c15a0725ace9a2f0a0a9e49"} Jul 17 18:42:47.856827 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:47.856822 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq" Jul 17 18:42:47.857160 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:47.856860 2572 scope.go:117] "RemoveContainer" containerID="44d4630d4ad58868d65474f2f15dac4af4a9801fdbc97984561b4218214713af" Jul 17 18:42:47.873278 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:47.873253 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq"] Jul 17 18:42:47.879822 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:47.879799 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-spfjq"] Jul 17 18:42:49.224822 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:49.224794 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0e81bcc-8565-4faf-bee4-623239646e71" path="/var/lib/kubelet/pods/a0e81bcc-8565-4faf-bee4-623239646e71/volumes" Jul 17 18:42:55.358316 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358281 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg"] Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358567 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="027da4ab-64eb-483b-ad13-737da15629f5" containerName="revoke-keys" Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358578 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="027da4ab-64eb-483b-ad13-737da15629f5" containerName="revoke-keys" Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358589 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0875077b-264b-41c3-9495-606a3c4736ce" containerName="revoke-keys" Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358595 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0875077b-264b-41c3-9495-606a3c4736ce" containerName="revoke-keys" Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358602 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e81bcc-8565-4faf-bee4-623239646e71" containerName="revoke-keys" Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358608 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e81bcc-8565-4faf-bee4-623239646e71" containerName="revoke-keys" Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358620 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0875077b-264b-41c3-9495-606a3c4736ce" containerName="revoke-keys" Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358626 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0875077b-264b-41c3-9495-606a3c4736ce" containerName="revoke-keys" Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358634 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e81bcc-8565-4faf-bee4-623239646e71" containerName="revoke-keys" Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358639 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e81bcc-8565-4faf-bee4-623239646e71" containerName="revoke-keys" Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358650 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="027da4ab-64eb-483b-ad13-737da15629f5" containerName="revoke-keys" Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358655 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="027da4ab-64eb-483b-ad13-737da15629f5" containerName="revoke-keys" Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358698 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0e81bcc-8565-4faf-bee4-623239646e71" containerName="revoke-keys" Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358705 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="027da4ab-64eb-483b-ad13-737da15629f5" containerName="revoke-keys" Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358711 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0875077b-264b-41c3-9495-606a3c4736ce" containerName="revoke-keys" Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358718 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="027da4ab-64eb-483b-ad13-737da15629f5" containerName="revoke-keys" Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358723 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0875077b-264b-41c3-9495-606a3c4736ce" containerName="revoke-keys" Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358728 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0875077b-264b-41c3-9495-606a3c4736ce" containerName="revoke-keys" Jul 17 18:42:55.358744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.358735 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0e81bcc-8565-4faf-bee4-623239646e71" containerName="revoke-keys" Jul 17 18:42:55.361405 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.361388 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg" Jul 17 18:42:55.365186 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.365164 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:42:55.365316 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.365190 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:42:55.372646 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.372620 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg"] Jul 17 18:42:55.431975 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.431943 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33d35fc3-f62b-4042-af78-e43b69a8cee5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg\" (UID: \"33d35fc3-f62b-4042-af78-e43b69a8cee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg" Jul 17 18:42:55.532800 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.532756 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33d35fc3-f62b-4042-af78-e43b69a8cee5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg\" (UID: \"33d35fc3-f62b-4042-af78-e43b69a8cee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg" Jul 17 18:42:55.533422 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.533404 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33d35fc3-f62b-4042-af78-e43b69a8cee5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg\" (UID: \"33d35fc3-f62b-4042-af78-e43b69a8cee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg" Jul 17 18:42:55.672307 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.672227 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg" Jul 17 18:42:55.710147 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.710119 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs"] Jul 17 18:42:55.710418 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.710406 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0875077b-264b-41c3-9495-606a3c4736ce" containerName="revoke-keys" Jul 17 18:42:55.710418 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.710419 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0875077b-264b-41c3-9495-606a3c4736ce" containerName="revoke-keys" Jul 17 18:42:55.710535 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.710431 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e81bcc-8565-4faf-bee4-623239646e71" containerName="revoke-keys" Jul 17 18:42:55.710535 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.710437 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e81bcc-8565-4faf-bee4-623239646e71" containerName="revoke-keys" Jul 17 18:42:55.710535 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.710452 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="027da4ab-64eb-483b-ad13-737da15629f5" containerName="revoke-keys" Jul 17 18:42:55.710535 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.710457 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="027da4ab-64eb-483b-ad13-737da15629f5" containerName="revoke-keys" Jul 17 18:42:55.710535 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.710516 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0e81bcc-8565-4faf-bee4-623239646e71" containerName="revoke-keys" Jul 17 18:42:55.710535 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.710524 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="027da4ab-64eb-483b-ad13-737da15629f5" containerName="revoke-keys" Jul 17 18:42:55.713641 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.713621 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:55.716563 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.716503 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3be95b\"/\"kube-root-ca.crt\"" Jul 17 18:42:55.716800 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.716780 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3be95b\"/\"test-model-3be95b-kserve-self-signed-certs\"" Jul 17 18:42:55.716917 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.716878 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3be95b\"/\"openshift-service-ca.crt\"" Jul 17 18:42:55.717038 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.716999 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3be95b\"/\"default-dockercfg-r9wn9\"" Jul 17 18:42:55.727619 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.727592 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs"] Jul 17 18:42:55.807730 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.807677 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg"] Jul 17 18:42:55.809892 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:42:55.809861 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d35fc3_f62b_4042_af78_e43b69a8cee5.slice/crio-a773faeb4009654aa404fe253ad38201f520d34d31ffae8b0e20a6f334dc612c WatchSource:0}: Error finding container a773faeb4009654aa404fe253ad38201f520d34d31ffae8b0e20a6f334dc612c: Status 404 returned error can't find the container with id a773faeb4009654aa404fe253ad38201f520d34d31ffae8b0e20a6f334dc612c Jul 17 18:42:55.835588 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.835563 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-model-cache\") pod \"test-model-3be95b-kserve-6577d47c97-jf4hs\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:55.835678 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.835594 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-tmp-dir\") pod \"test-model-3be95b-kserve-6577d47c97-jf4hs\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:55.835760 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.835686 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a1de4a-8d93-42be-814f-c8e9e924d1ae-tls-certs\") pod \"test-model-3be95b-kserve-6577d47c97-jf4hs\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:55.835760 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.835724 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-home\") pod \"test-model-3be95b-kserve-6577d47c97-jf4hs\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:55.835760 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.835747 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-dshm\") pod \"test-model-3be95b-kserve-6577d47c97-jf4hs\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:55.835861 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.835767 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkqsw\" (UniqueName: \"kubernetes.io/projected/28a1de4a-8d93-42be-814f-c8e9e924d1ae-kube-api-access-jkqsw\") pod \"test-model-3be95b-kserve-6577d47c97-jf4hs\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:55.890216 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.890190 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg" event={"ID":"33d35fc3-f62b-4042-af78-e43b69a8cee5","Type":"ContainerStarted","Data":"a773faeb4009654aa404fe253ad38201f520d34d31ffae8b0e20a6f334dc612c"} Jul 17 18:42:55.936381 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.936313 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-model-cache\") pod \"test-model-3be95b-kserve-6577d47c97-jf4hs\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:55.936381 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.936346 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-tmp-dir\") pod \"test-model-3be95b-kserve-6577d47c97-jf4hs\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:55.936549 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.936412 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a1de4a-8d93-42be-814f-c8e9e924d1ae-tls-certs\") pod \"test-model-3be95b-kserve-6577d47c97-jf4hs\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:55.936549 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.936447 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-home\") pod \"test-model-3be95b-kserve-6577d47c97-jf4hs\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:55.936549 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.936476 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-dshm\") pod \"test-model-3be95b-kserve-6577d47c97-jf4hs\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:55.936549 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.936498 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkqsw\" (UniqueName: \"kubernetes.io/projected/28a1de4a-8d93-42be-814f-c8e9e924d1ae-kube-api-access-jkqsw\") pod \"test-model-3be95b-kserve-6577d47c97-jf4hs\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:55.936828 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.936790 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-model-cache\") pod \"test-model-3be95b-kserve-6577d47c97-jf4hs\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:55.936914 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.936811 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-home\") pod \"test-model-3be95b-kserve-6577d47c97-jf4hs\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:55.936968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.936947 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-tmp-dir\") pod \"test-model-3be95b-kserve-6577d47c97-jf4hs\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:55.939154 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.939132 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-dshm\") pod \"test-model-3be95b-kserve-6577d47c97-jf4hs\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:55.939354 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.939334 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a1de4a-8d93-42be-814f-c8e9e924d1ae-tls-certs\") pod \"test-model-3be95b-kserve-6577d47c97-jf4hs\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:55.953149 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:55.953128 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkqsw\" (UniqueName: \"kubernetes.io/projected/28a1de4a-8d93-42be-814f-c8e9e924d1ae-kube-api-access-jkqsw\") pod \"test-model-3be95b-kserve-6577d47c97-jf4hs\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:56.025336 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:56.025298 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:42:56.151074 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:56.150883 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs"] Jul 17 18:42:56.153488 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:42:56.153448 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28a1de4a_8d93_42be_814f_c8e9e924d1ae.slice/crio-a060e3fb40fa79e3a4f2829d919382c8585fcaaae082d970f20d8c836d714300 WatchSource:0}: Error finding container a060e3fb40fa79e3a4f2829d919382c8585fcaaae082d970f20d8c836d714300: Status 404 returned error can't find the container with id a060e3fb40fa79e3a4f2829d919382c8585fcaaae082d970f20d8c836d714300 Jul 17 18:42:56.895791 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:56.895759 2572 generic.go:356] "Generic (PLEG): container finished" podID="33d35fc3-f62b-4042-af78-e43b69a8cee5" containerID="6b3b4be1b351933deb2d5f7c57c09b2e1c658c5af4613b7223af6e2bc5906538" exitCode=6 Jul 17 18:42:56.896286 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:56.895807 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg" event={"ID":"33d35fc3-f62b-4042-af78-e43b69a8cee5","Type":"ContainerDied","Data":"6b3b4be1b351933deb2d5f7c57c09b2e1c658c5af4613b7223af6e2bc5906538"} Jul 17 18:42:56.896286 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:56.896071 2572 scope.go:117] "RemoveContainer" containerID="6b3b4be1b351933deb2d5f7c57c09b2e1c658c5af4613b7223af6e2bc5906538" Jul 17 18:42:56.897395 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:56.897370 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" event={"ID":"28a1de4a-8d93-42be-814f-c8e9e924d1ae","Type":"ContainerStarted","Data":"9b036d0d27339e02b8d9baf0e1b2f59b6fbe85e08764fc0ec2a1ac2a5a3bc90c"} Jul 17 18:42:56.897501 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:56.897411 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" event={"ID":"28a1de4a-8d93-42be-814f-c8e9e924d1ae","Type":"ContainerStarted","Data":"a060e3fb40fa79e3a4f2829d919382c8585fcaaae082d970f20d8c836d714300"} Jul 17 18:42:56.926339 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:56.926297 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" podStartSLOduration=1.926283063 podStartE2EDuration="1.926283063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:42:56.925279164 +0000 UTC m=+2772.334706771" watchObservedRunningTime="2026-07-17 18:42:56.926283063 +0000 UTC m=+2772.335710668" Jul 17 18:42:57.903666 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:57.903575 2572 generic.go:356] "Generic (PLEG): container finished" podID="33d35fc3-f62b-4042-af78-e43b69a8cee5" containerID="3f73a63d5397bb30bd6d838e0dba4d2947087c35be343728ec3458eefa697ffb" exitCode=6 Jul 17 18:42:57.903666 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:57.903646 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg" event={"ID":"33d35fc3-f62b-4042-af78-e43b69a8cee5","Type":"ContainerDied","Data":"3f73a63d5397bb30bd6d838e0dba4d2947087c35be343728ec3458eefa697ffb"} Jul 17 18:42:57.904209 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:57.903705 2572 scope.go:117] "RemoveContainer" containerID="6b3b4be1b351933deb2d5f7c57c09b2e1c658c5af4613b7223af6e2bc5906538" Jul 17 18:42:57.904209 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:57.903927 2572 scope.go:117] "RemoveContainer" containerID="3f73a63d5397bb30bd6d838e0dba4d2947087c35be343728ec3458eefa697ffb" Jul 17 18:42:57.904209 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:42:57.904173 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg_odh-ai-gateway-infra(33d35fc3-f62b-4042-af78-e43b69a8cee5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg" podUID="33d35fc3-f62b-4042-af78-e43b69a8cee5" Jul 17 18:42:58.910630 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:42:58.910601 2572 scope.go:117] "RemoveContainer" containerID="3f73a63d5397bb30bd6d838e0dba4d2947087c35be343728ec3458eefa697ffb" Jul 17 18:42:58.911048 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:42:58.910788 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg_odh-ai-gateway-infra(33d35fc3-f62b-4042-af78-e43b69a8cee5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg" podUID="33d35fc3-f62b-4042-af78-e43b69a8cee5" Jul 17 18:43:06.026136 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:06.026055 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:43:06.026504 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:06.026310 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:43:06.039202 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:06.039178 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:43:06.955032 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:06.954988 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:43:07.483533 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:07.483502 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74"] Jul 17 18:43:07.487861 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:07.487842 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74" Jul 17 18:43:07.495308 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:07.495279 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74"] Jul 17 18:43:07.643110 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:07.643077 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b05ed68a-e7e0-4e14-862d-0b3b585fe553-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74\" (UID: \"b05ed68a-e7e0-4e14-862d-0b3b585fe553\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74" Jul 17 18:43:07.743763 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:07.743687 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b05ed68a-e7e0-4e14-862d-0b3b585fe553-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74\" (UID: \"b05ed68a-e7e0-4e14-862d-0b3b585fe553\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74" Jul 17 18:43:07.744271 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:07.744252 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b05ed68a-e7e0-4e14-862d-0b3b585fe553-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74\" (UID: \"b05ed68a-e7e0-4e14-862d-0b3b585fe553\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74" Jul 17 18:43:07.798894 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:07.798862 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74" Jul 17 18:43:08.129531 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:08.129505 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74"] Jul 17 18:43:08.131536 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:43:08.131514 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05ed68a_e7e0_4e14_862d_0b3b585fe553.slice/crio-483b1a386a7ba4b2b1c15000cfebf8976c2228e02ff48e2450e1762729f635fc WatchSource:0}: Error finding container 483b1a386a7ba4b2b1c15000cfebf8976c2228e02ff48e2450e1762729f635fc: Status 404 returned error can't find the container with id 483b1a386a7ba4b2b1c15000cfebf8976c2228e02ff48e2450e1762729f635fc Jul 17 18:43:08.952818 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:08.952785 2572 generic.go:356] "Generic (PLEG): container finished" podID="b05ed68a-e7e0-4e14-862d-0b3b585fe553" containerID="1dadd03201d0fd94bd9ee992e698cc15cdb2daece3373d0a91ee0ce0ad7eddbb" exitCode=6 Jul 17 18:43:08.953194 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:08.952861 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74" event={"ID":"b05ed68a-e7e0-4e14-862d-0b3b585fe553","Type":"ContainerDied","Data":"1dadd03201d0fd94bd9ee992e698cc15cdb2daece3373d0a91ee0ce0ad7eddbb"} Jul 17 18:43:08.953194 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:08.952909 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74" event={"ID":"b05ed68a-e7e0-4e14-862d-0b3b585fe553","Type":"ContainerStarted","Data":"483b1a386a7ba4b2b1c15000cfebf8976c2228e02ff48e2450e1762729f635fc"} Jul 17 18:43:08.953194 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:08.953053 2572 scope.go:117] "RemoveContainer" containerID="1dadd03201d0fd94bd9ee992e698cc15cdb2daece3373d0a91ee0ce0ad7eddbb" Jul 17 18:43:09.958296 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:09.958210 2572 generic.go:356] "Generic (PLEG): container finished" podID="b05ed68a-e7e0-4e14-862d-0b3b585fe553" containerID="a31e04d6478cd4127d147aaa2a7e5ad59b335ccad8fd3020ba6e7cf40e9f962c" exitCode=6 Jul 17 18:43:09.958734 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:09.958303 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74" event={"ID":"b05ed68a-e7e0-4e14-862d-0b3b585fe553","Type":"ContainerDied","Data":"a31e04d6478cd4127d147aaa2a7e5ad59b335ccad8fd3020ba6e7cf40e9f962c"} Jul 17 18:43:09.958734 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:09.958369 2572 scope.go:117] "RemoveContainer" containerID="1dadd03201d0fd94bd9ee992e698cc15cdb2daece3373d0a91ee0ce0ad7eddbb" Jul 17 18:43:09.958734 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:09.958464 2572 scope.go:117] "RemoveContainer" containerID="a31e04d6478cd4127d147aaa2a7e5ad59b335ccad8fd3020ba6e7cf40e9f962c" Jul 17 18:43:09.958734 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:43:09.958670 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74_odh-ai-gateway-infra(b05ed68a-e7e0-4e14-862d-0b3b585fe553)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74" podUID="b05ed68a-e7e0-4e14-862d-0b3b585fe553" Jul 17 18:43:10.964600 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:10.964572 2572 scope.go:117] "RemoveContainer" containerID="a31e04d6478cd4127d147aaa2a7e5ad59b335ccad8fd3020ba6e7cf40e9f962c" Jul 17 18:43:10.964978 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:43:10.964751 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74_odh-ai-gateway-infra(b05ed68a-e7e0-4e14-862d-0b3b585fe553)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74" podUID="b05ed68a-e7e0-4e14-862d-0b3b585fe553" Jul 17 18:43:12.219504 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:12.219473 2572 scope.go:117] "RemoveContainer" containerID="3f73a63d5397bb30bd6d838e0dba4d2947087c35be343728ec3458eefa697ffb" Jul 17 18:43:12.976572 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:12.976541 2572 generic.go:356] "Generic (PLEG): container finished" podID="33d35fc3-f62b-4042-af78-e43b69a8cee5" containerID="50e216df3fe3d0340ec1838baa5453ac4db717e09ee022b8525a60234df69a8a" exitCode=6 Jul 17 18:43:12.976741 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:12.976617 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg" event={"ID":"33d35fc3-f62b-4042-af78-e43b69a8cee5","Type":"ContainerDied","Data":"50e216df3fe3d0340ec1838baa5453ac4db717e09ee022b8525a60234df69a8a"} Jul 17 18:43:12.976741 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:12.976685 2572 scope.go:117] "RemoveContainer" containerID="3f73a63d5397bb30bd6d838e0dba4d2947087c35be343728ec3458eefa697ffb" Jul 17 18:43:12.977037 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:12.977002 2572 scope.go:117] "RemoveContainer" containerID="50e216df3fe3d0340ec1838baa5453ac4db717e09ee022b8525a60234df69a8a" Jul 17 18:43:12.977258 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:43:12.977239 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg_odh-ai-gateway-infra(33d35fc3-f62b-4042-af78-e43b69a8cee5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg" podUID="33d35fc3-f62b-4042-af78-e43b69a8cee5" Jul 17 18:43:13.507397 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:13.507361 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx"] Jul 17 18:43:13.511744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:13.511727 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx" Jul 17 18:43:13.517753 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:13.517712 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx"] Jul 17 18:43:13.591913 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:13.591871 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a94ffda-3080-47f9-b1b9-8aa3d19416b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx\" (UID: \"7a94ffda-3080-47f9-b1b9-8aa3d19416b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx" Jul 17 18:43:13.693428 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:13.693378 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a94ffda-3080-47f9-b1b9-8aa3d19416b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx\" (UID: \"7a94ffda-3080-47f9-b1b9-8aa3d19416b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx" Jul 17 18:43:13.694028 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:13.693990 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a94ffda-3080-47f9-b1b9-8aa3d19416b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx\" (UID: \"7a94ffda-3080-47f9-b1b9-8aa3d19416b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx" Jul 17 18:43:13.822951 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:13.822912 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx" Jul 17 18:43:13.948544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:13.948508 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx"] Jul 17 18:43:13.950681 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:43:13.950650 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a94ffda_3080_47f9_b1b9_8aa3d19416b7.slice/crio-2d8f823348674dd3f070fdd17d3485ed85b9b674ac1a41f78b316f7408cfdd93 WatchSource:0}: Error finding container 2d8f823348674dd3f070fdd17d3485ed85b9b674ac1a41f78b316f7408cfdd93: Status 404 returned error can't find the container with id 2d8f823348674dd3f070fdd17d3485ed85b9b674ac1a41f78b316f7408cfdd93 Jul 17 18:43:13.983186 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:13.983161 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx" event={"ID":"7a94ffda-3080-47f9-b1b9-8aa3d19416b7","Type":"ContainerStarted","Data":"2d8f823348674dd3f070fdd17d3485ed85b9b674ac1a41f78b316f7408cfdd93"} Jul 17 18:43:14.002213 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:14.002188 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg"] Jul 17 18:43:14.130818 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:14.130794 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg" Jul 17 18:43:14.197730 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:14.197706 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33d35fc3-f62b-4042-af78-e43b69a8cee5-maas-api-service-ca\") pod \"33d35fc3-f62b-4042-af78-e43b69a8cee5\" (UID: \"33d35fc3-f62b-4042-af78-e43b69a8cee5\") " Jul 17 18:43:14.198063 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:14.198043 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d35fc3-f62b-4042-af78-e43b69a8cee5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33d35fc3-f62b-4042-af78-e43b69a8cee5" (UID: "33d35fc3-f62b-4042-af78-e43b69a8cee5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:43:14.299302 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:14.299272 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33d35fc3-f62b-4042-af78-e43b69a8cee5-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:43:14.989200 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:14.989168 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg" Jul 17 18:43:14.989660 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:14.989165 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg" event={"ID":"33d35fc3-f62b-4042-af78-e43b69a8cee5","Type":"ContainerDied","Data":"a773faeb4009654aa404fe253ad38201f520d34d31ffae8b0e20a6f334dc612c"} Jul 17 18:43:14.989660 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:14.989297 2572 scope.go:117] "RemoveContainer" containerID="50e216df3fe3d0340ec1838baa5453ac4db717e09ee022b8525a60234df69a8a" Jul 17 18:43:14.990722 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:14.990696 2572 generic.go:356] "Generic (PLEG): container finished" podID="7a94ffda-3080-47f9-b1b9-8aa3d19416b7" containerID="e1fd79da395a52df99d39bd4b1739c5260adb03a76cde2fda2ca3ac7f58d5323" exitCode=6 Jul 17 18:43:14.990824 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:14.990800 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx" event={"ID":"7a94ffda-3080-47f9-b1b9-8aa3d19416b7","Type":"ContainerDied","Data":"e1fd79da395a52df99d39bd4b1739c5260adb03a76cde2fda2ca3ac7f58d5323"} Jul 17 18:43:14.990962 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:14.990949 2572 scope.go:117] "RemoveContainer" containerID="e1fd79da395a52df99d39bd4b1739c5260adb03a76cde2fda2ca3ac7f58d5323" Jul 17 18:43:15.020253 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:15.020225 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg"] Jul 17 18:43:15.024392 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:15.024368 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-7pqbg"] Jul 17 18:43:15.224497 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:15.224464 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33d35fc3-f62b-4042-af78-e43b69a8cee5" path="/var/lib/kubelet/pods/33d35fc3-f62b-4042-af78-e43b69a8cee5/volumes" Jul 17 18:43:15.997830 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:15.997733 2572 generic.go:356] "Generic (PLEG): container finished" podID="7a94ffda-3080-47f9-b1b9-8aa3d19416b7" containerID="fd9341c6500b8032c397f9dfc1b80b29d0504059a174af81df844e7f73848977" exitCode=6 Jul 17 18:43:15.998308 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:15.997825 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx" event={"ID":"7a94ffda-3080-47f9-b1b9-8aa3d19416b7","Type":"ContainerDied","Data":"fd9341c6500b8032c397f9dfc1b80b29d0504059a174af81df844e7f73848977"} Jul 17 18:43:15.998308 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:15.997883 2572 scope.go:117] "RemoveContainer" containerID="e1fd79da395a52df99d39bd4b1739c5260adb03a76cde2fda2ca3ac7f58d5323" Jul 17 18:43:15.998308 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:15.998130 2572 scope.go:117] "RemoveContainer" containerID="fd9341c6500b8032c397f9dfc1b80b29d0504059a174af81df844e7f73848977" Jul 17 18:43:15.998450 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:43:15.998323 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx_odh-ai-gateway-infra(7a94ffda-3080-47f9-b1b9-8aa3d19416b7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx" podUID="7a94ffda-3080-47f9-b1b9-8aa3d19416b7" Jul 17 18:43:17.006791 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:17.006762 2572 scope.go:117] "RemoveContainer" containerID="fd9341c6500b8032c397f9dfc1b80b29d0504059a174af81df844e7f73848977" Jul 17 18:43:17.007183 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:43:17.006964 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx_odh-ai-gateway-infra(7a94ffda-3080-47f9-b1b9-8aa3d19416b7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx" podUID="7a94ffda-3080-47f9-b1b9-8aa3d19416b7" Jul 17 18:43:18.379003 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:18.378971 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs"] Jul 17 18:43:18.508421 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:18.508388 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddeacf-5x5vt"] Jul 17 18:43:18.508712 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:18.508697 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33d35fc3-f62b-4042-af78-e43b69a8cee5" containerName="revoke-keys" Jul 17 18:43:18.508712 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:18.508712 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="33d35fc3-f62b-4042-af78-e43b69a8cee5" containerName="revoke-keys" Jul 17 18:43:18.508796 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:18.508722 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33d35fc3-f62b-4042-af78-e43b69a8cee5" containerName="revoke-keys" Jul 17 18:43:18.508796 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:18.508729 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="33d35fc3-f62b-4042-af78-e43b69a8cee5" containerName="revoke-keys" Jul 17 18:43:18.508796 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:18.508739 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33d35fc3-f62b-4042-af78-e43b69a8cee5" containerName="revoke-keys" Jul 17 18:43:18.508796 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:18.508745 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="33d35fc3-f62b-4042-af78-e43b69a8cee5" containerName="revoke-keys" Jul 17 18:43:18.508930 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:18.508799 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="33d35fc3-f62b-4042-af78-e43b69a8cee5" containerName="revoke-keys" Jul 17 18:43:18.508930 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:18.508808 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="33d35fc3-f62b-4042-af78-e43b69a8cee5" containerName="revoke-keys" Jul 17 18:43:18.513017 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:18.512989 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddeacf-5x5vt" Jul 17 18:43:18.520293 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:18.520024 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddeacf-5x5vt"] Jul 17 18:43:18.640771 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:18.640691 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdeee194-26c7-4e9e-bf37-7952198f083e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddeacf-5x5vt\" (UID: \"bdeee194-26c7-4e9e-bf37-7952198f083e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddeacf-5x5vt" Jul 17 18:43:18.741281 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:18.741251 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdeee194-26c7-4e9e-bf37-7952198f083e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddeacf-5x5vt\" (UID: \"bdeee194-26c7-4e9e-bf37-7952198f083e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddeacf-5x5vt" Jul 17 18:43:18.741889 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:18.741872 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdeee194-26c7-4e9e-bf37-7952198f083e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ddeacf-5x5vt\" (UID: \"bdeee194-26c7-4e9e-bf37-7952198f083e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddeacf-5x5vt" Jul 17 18:43:18.824487 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:18.824453 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddeacf-5x5vt" Jul 17 18:43:18.950123 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:18.950091 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddeacf-5x5vt"] Jul 17 18:43:18.953291 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:43:18.953261 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdeee194_26c7_4e9e_bf37_7952198f083e.slice/crio-41f8fa6f93ee75f1206a2707d7dd706deefa9e05c3bfcb8c71b76924f143d242 WatchSource:0}: Error finding container 41f8fa6f93ee75f1206a2707d7dd706deefa9e05c3bfcb8c71b76924f143d242: Status 404 returned error can't find the container with id 41f8fa6f93ee75f1206a2707d7dd706deefa9e05c3bfcb8c71b76924f143d242 Jul 17 18:43:19.015273 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:19.015237 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddeacf-5x5vt" event={"ID":"bdeee194-26c7-4e9e-bf37-7952198f083e","Type":"ContainerStarted","Data":"41f8fa6f93ee75f1206a2707d7dd706deefa9e05c3bfcb8c71b76924f143d242"} Jul 17 18:43:20.021239 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:20.021143 2572 generic.go:356] "Generic (PLEG): container finished" podID="bdeee194-26c7-4e9e-bf37-7952198f083e" containerID="35d49f9252926b2ae28ea382e9e69ff3edff3da4bb5e94b7cc3820bbdb2c29c1" exitCode=0 Jul 17 18:43:20.021239 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:20.021210 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddeacf-5x5vt" event={"ID":"bdeee194-26c7-4e9e-bf37-7952198f083e","Type":"ContainerDied","Data":"35d49f9252926b2ae28ea382e9e69ff3edff3da4bb5e94b7cc3820bbdb2c29c1"} Jul 17 18:43:20.964801 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:20.964764 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp"] Jul 17 18:43:20.965147 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:20.965135 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="33d35fc3-f62b-4042-af78-e43b69a8cee5" containerName="revoke-keys" Jul 17 18:43:20.968120 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:20.968102 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp" Jul 17 18:43:20.975617 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:20.975164 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp"] Jul 17 18:43:21.061673 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:21.061641 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a47fe6-5ef3-4849-9478-bbdde6252e39-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp\" (UID: \"50a47fe6-5ef3-4849-9478-bbdde6252e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp" Jul 17 18:43:21.151913 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:21.151886 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddeacf-5x5vt" Jul 17 18:43:21.162392 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:21.162369 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a47fe6-5ef3-4849-9478-bbdde6252e39-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp\" (UID: \"50a47fe6-5ef3-4849-9478-bbdde6252e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp" Jul 17 18:43:21.162940 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:21.162922 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a47fe6-5ef3-4849-9478-bbdde6252e39-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp\" (UID: \"50a47fe6-5ef3-4849-9478-bbdde6252e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp" Jul 17 18:43:21.263705 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:21.263610 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdeee194-26c7-4e9e-bf37-7952198f083e-maas-api-service-ca\") pod \"bdeee194-26c7-4e9e-bf37-7952198f083e\" (UID: \"bdeee194-26c7-4e9e-bf37-7952198f083e\") " Jul 17 18:43:21.264041 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:21.263994 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdeee194-26c7-4e9e-bf37-7952198f083e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bdeee194-26c7-4e9e-bf37-7952198f083e" (UID: "bdeee194-26c7-4e9e-bf37-7952198f083e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:43:21.280625 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:21.280597 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp" Jul 17 18:43:21.364806 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:21.364773 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdeee194-26c7-4e9e-bf37-7952198f083e-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:43:21.412391 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:21.412364 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp"] Jul 17 18:43:21.414778 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:43:21.414751 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a47fe6_5ef3_4849_9478_bbdde6252e39.slice/crio-416bb8f61c1bd9c2bac915ed3c42ee6a0db12b3337c1fdcb85492999beb453a7 WatchSource:0}: Error finding container 416bb8f61c1bd9c2bac915ed3c42ee6a0db12b3337c1fdcb85492999beb453a7: Status 404 returned error can't find the container with id 416bb8f61c1bd9c2bac915ed3c42ee6a0db12b3337c1fdcb85492999beb453a7 Jul 17 18:43:22.031751 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:22.031712 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp" event={"ID":"50a47fe6-5ef3-4849-9478-bbdde6252e39","Type":"ContainerStarted","Data":"61990bd1a9fb0b4f3833b1ba2c954fc14dfb936ae181c34182d8a420c212102b"} Jul 17 18:43:22.031950 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:22.031805 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp" event={"ID":"50a47fe6-5ef3-4849-9478-bbdde6252e39","Type":"ContainerStarted","Data":"416bb8f61c1bd9c2bac915ed3c42ee6a0db12b3337c1fdcb85492999beb453a7"} Jul 17 18:43:22.033093 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:22.033065 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddeacf-5x5vt" event={"ID":"bdeee194-26c7-4e9e-bf37-7952198f083e","Type":"ContainerDied","Data":"41f8fa6f93ee75f1206a2707d7dd706deefa9e05c3bfcb8c71b76924f143d242"} Jul 17 18:43:22.033196 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:22.033114 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41f8fa6f93ee75f1206a2707d7dd706deefa9e05c3bfcb8c71b76924f143d242" Jul 17 18:43:22.033196 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:22.033078 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ddeacf-5x5vt" Jul 17 18:43:22.048659 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:22.048618 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp" podStartSLOduration=2.048603894 podStartE2EDuration="2.048603894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:43:22.046904632 +0000 UTC m=+2797.456332238" watchObservedRunningTime="2026-07-17 18:43:22.048603894 +0000 UTC m=+2797.458031586" Jul 17 18:43:22.219513 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:22.219485 2572 scope.go:117] "RemoveContainer" containerID="a31e04d6478cd4127d147aaa2a7e5ad59b335ccad8fd3020ba6e7cf40e9f962c" Jul 17 18:43:23.038432 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:23.038400 2572 generic.go:356] "Generic (PLEG): container finished" podID="50a47fe6-5ef3-4849-9478-bbdde6252e39" containerID="61990bd1a9fb0b4f3833b1ba2c954fc14dfb936ae181c34182d8a420c212102b" exitCode=6 Jul 17 18:43:23.038613 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:23.038474 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp" event={"ID":"50a47fe6-5ef3-4849-9478-bbdde6252e39","Type":"ContainerDied","Data":"61990bd1a9fb0b4f3833b1ba2c954fc14dfb936ae181c34182d8a420c212102b"} Jul 17 18:43:23.038685 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:23.038669 2572 scope.go:117] "RemoveContainer" containerID="61990bd1a9fb0b4f3833b1ba2c954fc14dfb936ae181c34182d8a420c212102b" Jul 17 18:43:23.040219 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:23.040199 2572 generic.go:356] "Generic (PLEG): container finished" podID="b05ed68a-e7e0-4e14-862d-0b3b585fe553" containerID="e17f918e59e15ad17eacfdbc54369f657fd03338be444b25d78d51f2acc523f4" exitCode=6 Jul 17 18:43:23.040314 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:23.040269 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74" event={"ID":"b05ed68a-e7e0-4e14-862d-0b3b585fe553","Type":"ContainerDied","Data":"e17f918e59e15ad17eacfdbc54369f657fd03338be444b25d78d51f2acc523f4"} Jul 17 18:43:23.040356 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:23.040316 2572 scope.go:117] "RemoveContainer" containerID="a31e04d6478cd4127d147aaa2a7e5ad59b335ccad8fd3020ba6e7cf40e9f962c" Jul 17 18:43:23.040533 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:23.040520 2572 scope.go:117] "RemoveContainer" containerID="e17f918e59e15ad17eacfdbc54369f657fd03338be444b25d78d51f2acc523f4" Jul 17 18:43:23.040781 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:43:23.040702 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74_odh-ai-gateway-infra(b05ed68a-e7e0-4e14-862d-0b3b585fe553)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74" podUID="b05ed68a-e7e0-4e14-862d-0b3b585fe553" Jul 17 18:43:24.047889 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:24.047859 2572 generic.go:356] "Generic (PLEG): container finished" podID="50a47fe6-5ef3-4849-9478-bbdde6252e39" containerID="0d062a0391767e0974e6c644993dfd8cf9ed8f9f101430734ce8ef09d31301fa" exitCode=6 Jul 17 18:43:24.048346 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:24.047894 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp" event={"ID":"50a47fe6-5ef3-4849-9478-bbdde6252e39","Type":"ContainerDied","Data":"0d062a0391767e0974e6c644993dfd8cf9ed8f9f101430734ce8ef09d31301fa"} Jul 17 18:43:24.048346 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:24.047936 2572 scope.go:117] "RemoveContainer" containerID="61990bd1a9fb0b4f3833b1ba2c954fc14dfb936ae181c34182d8a420c212102b" Jul 17 18:43:24.048346 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:24.048177 2572 scope.go:117] "RemoveContainer" containerID="0d062a0391767e0974e6c644993dfd8cf9ed8f9f101430734ce8ef09d31301fa" Jul 17 18:43:24.048485 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:43:24.048356 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp_odh-ai-gateway-infra(50a47fe6-5ef3-4849-9478-bbdde6252e39)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp" podUID="50a47fe6-5ef3-4849-9478-bbdde6252e39" Jul 17 18:43:24.078207 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:24.078173 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74"] Jul 17 18:43:24.228786 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:24.228757 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74" Jul 17 18:43:24.393080 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:24.392966 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b05ed68a-e7e0-4e14-862d-0b3b585fe553-maas-api-service-ca\") pod \"b05ed68a-e7e0-4e14-862d-0b3b585fe553\" (UID: \"b05ed68a-e7e0-4e14-862d-0b3b585fe553\") " Jul 17 18:43:24.393308 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:24.393285 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b05ed68a-e7e0-4e14-862d-0b3b585fe553-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b05ed68a-e7e0-4e14-862d-0b3b585fe553" (UID: "b05ed68a-e7e0-4e14-862d-0b3b585fe553"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:43:24.493600 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:24.493558 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b05ed68a-e7e0-4e14-862d-0b3b585fe553-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:43:25.053942 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:25.053911 2572 scope.go:117] "RemoveContainer" containerID="0d062a0391767e0974e6c644993dfd8cf9ed8f9f101430734ce8ef09d31301fa" Jul 17 18:43:25.054388 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:43:25.054137 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp_odh-ai-gateway-infra(50a47fe6-5ef3-4849-9478-bbdde6252e39)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp" podUID="50a47fe6-5ef3-4849-9478-bbdde6252e39" Jul 17 18:43:25.054833 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:25.054813 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74" event={"ID":"b05ed68a-e7e0-4e14-862d-0b3b585fe553","Type":"ContainerDied","Data":"483b1a386a7ba4b2b1c15000cfebf8976c2228e02ff48e2450e1762729f635fc"} Jul 17 18:43:25.054887 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:25.054816 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74" Jul 17 18:43:25.054943 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:25.054862 2572 scope.go:117] "RemoveContainer" containerID="e17f918e59e15ad17eacfdbc54369f657fd03338be444b25d78d51f2acc523f4" Jul 17 18:43:25.081448 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:25.080875 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74"] Jul 17 18:43:25.082754 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:25.082730 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-hjz74"] Jul 17 18:43:25.225127 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:25.225091 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b05ed68a-e7e0-4e14-862d-0b3b585fe553" path="/var/lib/kubelet/pods/b05ed68a-e7e0-4e14-862d-0b3b585fe553/volumes" Jul 17 18:43:28.219748 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:28.219715 2572 scope.go:117] "RemoveContainer" containerID="fd9341c6500b8032c397f9dfc1b80b29d0504059a174af81df844e7f73848977" Jul 17 18:43:29.078932 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:29.078900 2572 generic.go:356] "Generic (PLEG): container finished" podID="7a94ffda-3080-47f9-b1b9-8aa3d19416b7" containerID="699fa309d6fb3ae083805be32d6ca4c1188b7b877a0881492dca188a4e8b8603" exitCode=6 Jul 17 18:43:29.079116 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:29.078969 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx" event={"ID":"7a94ffda-3080-47f9-b1b9-8aa3d19416b7","Type":"ContainerDied","Data":"699fa309d6fb3ae083805be32d6ca4c1188b7b877a0881492dca188a4e8b8603"} Jul 17 18:43:29.079116 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:29.079048 2572 scope.go:117] "RemoveContainer" containerID="fd9341c6500b8032c397f9dfc1b80b29d0504059a174af81df844e7f73848977" Jul 17 18:43:29.079388 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:29.079368 2572 scope.go:117] "RemoveContainer" containerID="699fa309d6fb3ae083805be32d6ca4c1188b7b877a0881492dca188a4e8b8603" Jul 17 18:43:29.079611 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:43:29.079589 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx_odh-ai-gateway-infra(7a94ffda-3080-47f9-b1b9-8aa3d19416b7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx" podUID="7a94ffda-3080-47f9-b1b9-8aa3d19416b7" Jul 17 18:43:30.103792 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:30.103760 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx"] Jul 17 18:43:30.234238 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:30.234213 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx" Jul 17 18:43:30.243040 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:30.243004 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a94ffda-3080-47f9-b1b9-8aa3d19416b7-maas-api-service-ca\") pod \"7a94ffda-3080-47f9-b1b9-8aa3d19416b7\" (UID: \"7a94ffda-3080-47f9-b1b9-8aa3d19416b7\") " Jul 17 18:43:30.243286 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:30.243266 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a94ffda-3080-47f9-b1b9-8aa3d19416b7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7a94ffda-3080-47f9-b1b9-8aa3d19416b7" (UID: "7a94ffda-3080-47f9-b1b9-8aa3d19416b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:43:30.344308 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:30.344278 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a94ffda-3080-47f9-b1b9-8aa3d19416b7-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:43:31.089995 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:31.089964 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx" event={"ID":"7a94ffda-3080-47f9-b1b9-8aa3d19416b7","Type":"ContainerDied","Data":"2d8f823348674dd3f070fdd17d3485ed85b9b674ac1a41f78b316f7408cfdd93"} Jul 17 18:43:31.089995 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:31.089995 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx" Jul 17 18:43:31.090273 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:31.090046 2572 scope.go:117] "RemoveContainer" containerID="699fa309d6fb3ae083805be32d6ca4c1188b7b877a0881492dca188a4e8b8603" Jul 17 18:43:31.110474 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:31.110449 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx"] Jul 17 18:43:31.117873 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:31.117847 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-lgwpx"] Jul 17 18:43:31.225704 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:31.225672 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a94ffda-3080-47f9-b1b9-8aa3d19416b7" path="/var/lib/kubelet/pods/7a94ffda-3080-47f9-b1b9-8aa3d19416b7/volumes" Jul 17 18:43:33.398847 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.398784 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" podUID="28a1de4a-8d93-42be-814f-c8e9e924d1ae" containerName="main" containerID="cri-o://9b036d0d27339e02b8d9baf0e1b2f59b6fbe85e08764fc0ec2a1ac2a5a3bc90c" gracePeriod=45 Jul 17 18:43:33.654316 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.654259 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:43:33.667144 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.667120 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a1de4a-8d93-42be-814f-c8e9e924d1ae-tls-certs\") pod \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " Jul 17 18:43:33.667229 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.667162 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-model-cache\") pod \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " Jul 17 18:43:33.667229 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.667188 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkqsw\" (UniqueName: \"kubernetes.io/projected/28a1de4a-8d93-42be-814f-c8e9e924d1ae-kube-api-access-jkqsw\") pod \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " Jul 17 18:43:33.667229 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.667221 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-home\") pod \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " Jul 17 18:43:33.667382 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.667239 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-dshm\") pod \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " Jul 17 18:43:33.667382 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.667266 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-tmp-dir\") pod \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\" (UID: \"28a1de4a-8d93-42be-814f-c8e9e924d1ae\") " Jul 17 18:43:33.667490 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.667417 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "28a1de4a-8d93-42be-814f-c8e9e924d1ae" (UID: "28a1de4a-8d93-42be-814f-c8e9e924d1ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:43:33.667548 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.667492 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-home" (OuterVolumeSpecName: "home") pod "28a1de4a-8d93-42be-814f-c8e9e924d1ae" (UID: "28a1de4a-8d93-42be-814f-c8e9e924d1ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:43:33.667548 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.667508 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-model-cache\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:43:33.667663 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.667570 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "28a1de4a-8d93-42be-814f-c8e9e924d1ae" (UID: "28a1de4a-8d93-42be-814f-c8e9e924d1ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:43:33.669332 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.669306 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-dshm" (OuterVolumeSpecName: "dshm") pod "28a1de4a-8d93-42be-814f-c8e9e924d1ae" (UID: "28a1de4a-8d93-42be-814f-c8e9e924d1ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:43:33.669433 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.669343 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a1de4a-8d93-42be-814f-c8e9e924d1ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28a1de4a-8d93-42be-814f-c8e9e924d1ae" (UID: "28a1de4a-8d93-42be-814f-c8e9e924d1ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:43:33.669483 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.669429 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a1de4a-8d93-42be-814f-c8e9e924d1ae-kube-api-access-jkqsw" (OuterVolumeSpecName: "kube-api-access-jkqsw") pod "28a1de4a-8d93-42be-814f-c8e9e924d1ae" (UID: "28a1de4a-8d93-42be-814f-c8e9e924d1ae"). InnerVolumeSpecName "kube-api-access-jkqsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:43:33.767952 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.767920 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-home\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:43:33.767952 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.767951 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-dshm\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:43:33.767952 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.767962 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a1de4a-8d93-42be-814f-c8e9e924d1ae-tmp-dir\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:43:33.768228 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.767972 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a1de4a-8d93-42be-814f-c8e9e924d1ae-tls-certs\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:43:33.768228 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:33.767984 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkqsw\" (UniqueName: \"kubernetes.io/projected/28a1de4a-8d93-42be-814f-c8e9e924d1ae-kube-api-access-jkqsw\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:43:34.104819 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:34.104781 2572 generic.go:356] "Generic (PLEG): container finished" podID="28a1de4a-8d93-42be-814f-c8e9e924d1ae" containerID="9b036d0d27339e02b8d9baf0e1b2f59b6fbe85e08764fc0ec2a1ac2a5a3bc90c" exitCode=0 Jul 17 18:43:34.105028 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:34.104829 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" Jul 17 18:43:34.105028 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:34.104827 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" event={"ID":"28a1de4a-8d93-42be-814f-c8e9e924d1ae","Type":"ContainerDied","Data":"9b036d0d27339e02b8d9baf0e1b2f59b6fbe85e08764fc0ec2a1ac2a5a3bc90c"} Jul 17 18:43:34.105028 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:34.104947 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs" event={"ID":"28a1de4a-8d93-42be-814f-c8e9e924d1ae","Type":"ContainerDied","Data":"a060e3fb40fa79e3a4f2829d919382c8585fcaaae082d970f20d8c836d714300"} Jul 17 18:43:34.105028 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:34.104972 2572 scope.go:117] "RemoveContainer" containerID="9b036d0d27339e02b8d9baf0e1b2f59b6fbe85e08764fc0ec2a1ac2a5a3bc90c" Jul 17 18:43:34.114337 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:34.113922 2572 scope.go:117] "RemoveContainer" containerID="9b036d0d27339e02b8d9baf0e1b2f59b6fbe85e08764fc0ec2a1ac2a5a3bc90c" Jul 17 18:43:34.114337 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:43:34.114250 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b036d0d27339e02b8d9baf0e1b2f59b6fbe85e08764fc0ec2a1ac2a5a3bc90c\": container with ID starting with 9b036d0d27339e02b8d9baf0e1b2f59b6fbe85e08764fc0ec2a1ac2a5a3bc90c not found: ID does not exist" containerID="9b036d0d27339e02b8d9baf0e1b2f59b6fbe85e08764fc0ec2a1ac2a5a3bc90c" Jul 17 18:43:34.114337 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:34.114282 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b036d0d27339e02b8d9baf0e1b2f59b6fbe85e08764fc0ec2a1ac2a5a3bc90c"} err="failed to get container status \"9b036d0d27339e02b8d9baf0e1b2f59b6fbe85e08764fc0ec2a1ac2a5a3bc90c\": rpc error: code = NotFound desc = could not find container \"9b036d0d27339e02b8d9baf0e1b2f59b6fbe85e08764fc0ec2a1ac2a5a3bc90c\": container with ID starting with 9b036d0d27339e02b8d9baf0e1b2f59b6fbe85e08764fc0ec2a1ac2a5a3bc90c not found: ID does not exist" Jul 17 18:43:34.128781 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:34.128755 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs"] Jul 17 18:43:34.134646 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:34.134624 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3be95b/test-model-3be95b-kserve-6577d47c97-jf4hs"] Jul 17 18:43:35.224613 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:35.224579 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28a1de4a-8d93-42be-814f-c8e9e924d1ae" path="/var/lib/kubelet/pods/28a1de4a-8d93-42be-814f-c8e9e924d1ae/volumes" Jul 17 18:43:36.220204 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:36.220175 2572 scope.go:117] "RemoveContainer" containerID="0d062a0391767e0974e6c644993dfd8cf9ed8f9f101430734ce8ef09d31301fa" Jul 17 18:43:37.120148 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:37.120068 2572 generic.go:356] "Generic (PLEG): container finished" podID="50a47fe6-5ef3-4849-9478-bbdde6252e39" containerID="03da255c2cb0ff676c073509bed2488a8dcff8fa49f3329bf24650dcb9795d07" exitCode=6 Jul 17 18:43:37.120148 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:37.120106 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp" event={"ID":"50a47fe6-5ef3-4849-9478-bbdde6252e39","Type":"ContainerDied","Data":"03da255c2cb0ff676c073509bed2488a8dcff8fa49f3329bf24650dcb9795d07"} Jul 17 18:43:37.120587 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:37.120160 2572 scope.go:117] "RemoveContainer" containerID="0d062a0391767e0974e6c644993dfd8cf9ed8f9f101430734ce8ef09d31301fa" Jul 17 18:43:37.120587 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:37.120440 2572 scope.go:117] "RemoveContainer" containerID="03da255c2cb0ff676c073509bed2488a8dcff8fa49f3329bf24650dcb9795d07" Jul 17 18:43:37.120695 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:43:37.120646 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp_odh-ai-gateway-infra(50a47fe6-5ef3-4849-9478-bbdde6252e39)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp" podUID="50a47fe6-5ef3-4849-9478-bbdde6252e39" Jul 17 18:43:38.146881 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:38.146851 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp"] Jul 17 18:43:38.278119 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:38.278096 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp" Jul 17 18:43:38.304975 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:38.304951 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a47fe6-5ef3-4849-9478-bbdde6252e39-maas-api-service-ca\") pod \"50a47fe6-5ef3-4849-9478-bbdde6252e39\" (UID: \"50a47fe6-5ef3-4849-9478-bbdde6252e39\") " Jul 17 18:43:38.305291 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:38.305269 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a47fe6-5ef3-4849-9478-bbdde6252e39-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50a47fe6-5ef3-4849-9478-bbdde6252e39" (UID: "50a47fe6-5ef3-4849-9478-bbdde6252e39"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:43:38.406287 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:38.406201 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a47fe6-5ef3-4849-9478-bbdde6252e39-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:43:39.131196 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:39.131163 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp" Jul 17 18:43:39.131403 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:39.131198 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp" event={"ID":"50a47fe6-5ef3-4849-9478-bbdde6252e39","Type":"ContainerDied","Data":"416bb8f61c1bd9c2bac915ed3c42ee6a0db12b3337c1fdcb85492999beb453a7"} Jul 17 18:43:39.131403 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:39.131266 2572 scope.go:117] "RemoveContainer" containerID="03da255c2cb0ff676c073509bed2488a8dcff8fa49f3329bf24650dcb9795d07" Jul 17 18:43:39.151292 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:39.151265 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp"] Jul 17 18:43:39.156806 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:39.156785 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7g9cp"] Jul 17 18:43:39.224267 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:39.224240 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50a47fe6-5ef3-4849-9478-bbdde6252e39" path="/var/lib/kubelet/pods/50a47fe6-5ef3-4849-9478-bbdde6252e39/volumes" Jul 17 18:43:44.682948 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.682916 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3be95b-dzvdl"] Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683206 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28a1de4a-8d93-42be-814f-c8e9e924d1ae" containerName="main" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683217 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="28a1de4a-8d93-42be-814f-c8e9e924d1ae" containerName="main" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683227 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50a47fe6-5ef3-4849-9478-bbdde6252e39" containerName="revoke-keys" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683232 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="50a47fe6-5ef3-4849-9478-bbdde6252e39" containerName="revoke-keys" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683242 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50a47fe6-5ef3-4849-9478-bbdde6252e39" containerName="revoke-keys" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683248 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="50a47fe6-5ef3-4849-9478-bbdde6252e39" containerName="revoke-keys" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683256 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50a47fe6-5ef3-4849-9478-bbdde6252e39" containerName="revoke-keys" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683261 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="50a47fe6-5ef3-4849-9478-bbdde6252e39" containerName="revoke-keys" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683269 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b05ed68a-e7e0-4e14-862d-0b3b585fe553" containerName="revoke-keys" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683275 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b05ed68a-e7e0-4e14-862d-0b3b585fe553" containerName="revoke-keys" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683286 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a94ffda-3080-47f9-b1b9-8aa3d19416b7" containerName="revoke-keys" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683291 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a94ffda-3080-47f9-b1b9-8aa3d19416b7" containerName="revoke-keys" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683300 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a94ffda-3080-47f9-b1b9-8aa3d19416b7" containerName="revoke-keys" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683305 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a94ffda-3080-47f9-b1b9-8aa3d19416b7" containerName="revoke-keys" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683311 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdeee194-26c7-4e9e-bf37-7952198f083e" containerName="revoke-keys" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683317 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdeee194-26c7-4e9e-bf37-7952198f083e" containerName="revoke-keys" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683324 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b05ed68a-e7e0-4e14-862d-0b3b585fe553" containerName="revoke-keys" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683329 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b05ed68a-e7e0-4e14-862d-0b3b585fe553" containerName="revoke-keys" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683334 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a94ffda-3080-47f9-b1b9-8aa3d19416b7" containerName="revoke-keys" Jul 17 18:43:44.683351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683340 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a94ffda-3080-47f9-b1b9-8aa3d19416b7" containerName="revoke-keys" Jul 17 18:43:44.683951 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683380 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="28a1de4a-8d93-42be-814f-c8e9e924d1ae" containerName="main" Jul 17 18:43:44.683951 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683387 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b05ed68a-e7e0-4e14-862d-0b3b585fe553" containerName="revoke-keys" Jul 17 18:43:44.683951 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683392 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a94ffda-3080-47f9-b1b9-8aa3d19416b7" containerName="revoke-keys" Jul 17 18:43:44.683951 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683398 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a94ffda-3080-47f9-b1b9-8aa3d19416b7" containerName="revoke-keys" Jul 17 18:43:44.683951 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683403 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a94ffda-3080-47f9-b1b9-8aa3d19416b7" containerName="revoke-keys" Jul 17 18:43:44.683951 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683409 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b05ed68a-e7e0-4e14-862d-0b3b585fe553" containerName="revoke-keys" Jul 17 18:43:44.683951 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683413 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="50a47fe6-5ef3-4849-9478-bbdde6252e39" containerName="revoke-keys" Jul 17 18:43:44.683951 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683418 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdeee194-26c7-4e9e-bf37-7952198f083e" containerName="revoke-keys" Jul 17 18:43:44.683951 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.683424 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="50a47fe6-5ef3-4849-9478-bbdde6252e39" containerName="revoke-keys" Jul 17 18:43:44.687932 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.687913 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3be95b-dzvdl" Jul 17 18:43:44.690274 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.690253 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:43:44.690406 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.690252 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:43:44.693444 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.693423 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3be95b-dzvdl"] Jul 17 18:43:44.758557 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.758525 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6648d623-a8f1-44bb-87d1-399f9bdf2d05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3be95b-dzvdl\" (UID: \"6648d623-a8f1-44bb-87d1-399f9bdf2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3be95b-dzvdl" Jul 17 18:43:44.859523 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.859482 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6648d623-a8f1-44bb-87d1-399f9bdf2d05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3be95b-dzvdl\" (UID: \"6648d623-a8f1-44bb-87d1-399f9bdf2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3be95b-dzvdl" Jul 17 18:43:44.860136 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.860113 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6648d623-a8f1-44bb-87d1-399f9bdf2d05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3be95b-dzvdl\" (UID: \"6648d623-a8f1-44bb-87d1-399f9bdf2d05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3be95b-dzvdl" Jul 17 18:43:44.999927 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:44.999844 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3be95b-dzvdl" Jul 17 18:43:45.132079 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:45.131992 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3be95b-dzvdl"] Jul 17 18:43:45.134350 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:43:45.134326 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6648d623_a8f1_44bb_87d1_399f9bdf2d05.slice/crio-820dcf1c9a7e2cb32715945d6c4e30ff4b266a9a3eb72a3f78e616a96b908275 WatchSource:0}: Error finding container 820dcf1c9a7e2cb32715945d6c4e30ff4b266a9a3eb72a3f78e616a96b908275: Status 404 returned error can't find the container with id 820dcf1c9a7e2cb32715945d6c4e30ff4b266a9a3eb72a3f78e616a96b908275 Jul 17 18:43:45.159861 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:45.159832 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3be95b-dzvdl" event={"ID":"6648d623-a8f1-44bb-87d1-399f9bdf2d05","Type":"ContainerStarted","Data":"820dcf1c9a7e2cb32715945d6c4e30ff4b266a9a3eb72a3f78e616a96b908275"} Jul 17 18:43:45.394396 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:45.394362 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg"] Jul 17 18:43:45.394691 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:45.394677 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b05ed68a-e7e0-4e14-862d-0b3b585fe553" containerName="revoke-keys" Jul 17 18:43:45.394738 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:45.394693 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b05ed68a-e7e0-4e14-862d-0b3b585fe553" containerName="revoke-keys" Jul 17 18:43:45.394775 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:45.394749 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="50a47fe6-5ef3-4849-9478-bbdde6252e39" containerName="revoke-keys" Jul 17 18:43:45.394775 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:45.394760 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b05ed68a-e7e0-4e14-862d-0b3b585fe553" containerName="revoke-keys" Jul 17 18:43:45.398826 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:45.398807 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg" Jul 17 18:43:45.404752 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:45.404723 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg"] Jul 17 18:43:45.466187 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:45.466156 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4829e7a8-c318-4e87-90a0-898702853415-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg\" (UID: \"4829e7a8-c318-4e87-90a0-898702853415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg" Jul 17 18:43:45.567392 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:45.567359 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4829e7a8-c318-4e87-90a0-898702853415-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg\" (UID: \"4829e7a8-c318-4e87-90a0-898702853415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg" Jul 17 18:43:45.567980 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:45.567960 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4829e7a8-c318-4e87-90a0-898702853415-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg\" (UID: \"4829e7a8-c318-4e87-90a0-898702853415\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg" Jul 17 18:43:45.709401 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:45.709322 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg" Jul 17 18:43:45.839662 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:45.839631 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg"] Jul 17 18:43:45.841241 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:43:45.841216 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4829e7a8_c318_4e87_90a0_898702853415.slice/crio-bf3f61b6aa40ac415428ea8ba411856dfd927a8e5766156259cce3edede9bfd3 WatchSource:0}: Error finding container bf3f61b6aa40ac415428ea8ba411856dfd927a8e5766156259cce3edede9bfd3: Status 404 returned error can't find the container with id bf3f61b6aa40ac415428ea8ba411856dfd927a8e5766156259cce3edede9bfd3 Jul 17 18:43:46.166522 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:46.166478 2572 generic.go:356] "Generic (PLEG): container finished" podID="6648d623-a8f1-44bb-87d1-399f9bdf2d05" containerID="f5805eda78ef173783c18b14f941216cfcbfceb17ba384738a65d2e5d856944e" exitCode=0 Jul 17 18:43:46.166688 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:46.166565 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3be95b-dzvdl" event={"ID":"6648d623-a8f1-44bb-87d1-399f9bdf2d05","Type":"ContainerDied","Data":"f5805eda78ef173783c18b14f941216cfcbfceb17ba384738a65d2e5d856944e"} Jul 17 18:43:46.168053 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:46.168027 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg" event={"ID":"4829e7a8-c318-4e87-90a0-898702853415","Type":"ContainerStarted","Data":"693303f16ee9df11c1c8770b57f43ec3556621f60890f507d7f78f68b28e2bf0"} Jul 17 18:43:46.168180 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:46.168072 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg" event={"ID":"4829e7a8-c318-4e87-90a0-898702853415","Type":"ContainerStarted","Data":"bf3f61b6aa40ac415428ea8ba411856dfd927a8e5766156259cce3edede9bfd3"} Jul 17 18:43:46.197354 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:46.197308 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg" podStartSLOduration=1.197294675 podStartE2EDuration="1.197294675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:43:46.196083875 +0000 UTC m=+2821.605511551" watchObservedRunningTime="2026-07-17 18:43:46.197294675 +0000 UTC m=+2821.606722281" Jul 17 18:43:47.174350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:47.174316 2572 generic.go:356] "Generic (PLEG): container finished" podID="4829e7a8-c318-4e87-90a0-898702853415" containerID="693303f16ee9df11c1c8770b57f43ec3556621f60890f507d7f78f68b28e2bf0" exitCode=6 Jul 17 18:43:47.174812 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:47.174400 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg" event={"ID":"4829e7a8-c318-4e87-90a0-898702853415","Type":"ContainerDied","Data":"693303f16ee9df11c1c8770b57f43ec3556621f60890f507d7f78f68b28e2bf0"} Jul 17 18:43:47.174812 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:47.174620 2572 scope.go:117] "RemoveContainer" containerID="693303f16ee9df11c1c8770b57f43ec3556621f60890f507d7f78f68b28e2bf0" Jul 17 18:43:47.303003 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:47.302981 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3be95b-dzvdl" Jul 17 18:43:47.384875 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:47.384846 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6648d623-a8f1-44bb-87d1-399f9bdf2d05-maas-api-service-ca\") pod \"6648d623-a8f1-44bb-87d1-399f9bdf2d05\" (UID: \"6648d623-a8f1-44bb-87d1-399f9bdf2d05\") " Jul 17 18:43:47.385172 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:47.385152 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6648d623-a8f1-44bb-87d1-399f9bdf2d05-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6648d623-a8f1-44bb-87d1-399f9bdf2d05" (UID: "6648d623-a8f1-44bb-87d1-399f9bdf2d05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:43:47.486361 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:47.486330 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6648d623-a8f1-44bb-87d1-399f9bdf2d05-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:43:48.181026 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:48.180980 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3be95b-dzvdl" Jul 17 18:43:48.181466 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:48.181020 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3be95b-dzvdl" event={"ID":"6648d623-a8f1-44bb-87d1-399f9bdf2d05","Type":"ContainerDied","Data":"820dcf1c9a7e2cb32715945d6c4e30ff4b266a9a3eb72a3f78e616a96b908275"} Jul 17 18:43:48.181466 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:48.181076 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="820dcf1c9a7e2cb32715945d6c4e30ff4b266a9a3eb72a3f78e616a96b908275" Jul 17 18:43:48.182663 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:48.182639 2572 generic.go:356] "Generic (PLEG): container finished" podID="4829e7a8-c318-4e87-90a0-898702853415" containerID="66d65dfb764b00900931da505b778feada647bafaf3bf3a604a3523bd6520b55" exitCode=6 Jul 17 18:43:48.182795 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:48.182725 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg" event={"ID":"4829e7a8-c318-4e87-90a0-898702853415","Type":"ContainerDied","Data":"66d65dfb764b00900931da505b778feada647bafaf3bf3a604a3523bd6520b55"} Jul 17 18:43:48.182795 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:48.182781 2572 scope.go:117] "RemoveContainer" containerID="693303f16ee9df11c1c8770b57f43ec3556621f60890f507d7f78f68b28e2bf0" Jul 17 18:43:48.182987 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:48.182973 2572 scope.go:117] "RemoveContainer" containerID="66d65dfb764b00900931da505b778feada647bafaf3bf3a604a3523bd6520b55" Jul 17 18:43:48.183232 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:43:48.183214 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg_odh-ai-gateway-infra(4829e7a8-c318-4e87-90a0-898702853415)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg" podUID="4829e7a8-c318-4e87-90a0-898702853415" Jul 17 18:43:49.189647 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:43:49.189621 2572 scope.go:117] "RemoveContainer" containerID="66d65dfb764b00900931da505b778feada647bafaf3bf3a604a3523bd6520b55" Jul 17 18:43:49.190170 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:43:49.189801 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg_odh-ai-gateway-infra(4829e7a8-c318-4e87-90a0-898702853415)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg" podUID="4829e7a8-c318-4e87-90a0-898702853415" Jul 17 18:44:04.220324 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:04.220297 2572 scope.go:117] "RemoveContainer" containerID="66d65dfb764b00900931da505b778feada647bafaf3bf3a604a3523bd6520b55" Jul 17 18:44:05.256265 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:05.256173 2572 generic.go:356] "Generic (PLEG): container finished" podID="4829e7a8-c318-4e87-90a0-898702853415" containerID="c83e1f010dc15dfc0a9434af4ac125af2809fcde215a255765ede0af3cef8687" exitCode=6 Jul 17 18:44:05.256265 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:05.256243 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg" event={"ID":"4829e7a8-c318-4e87-90a0-898702853415","Type":"ContainerDied","Data":"c83e1f010dc15dfc0a9434af4ac125af2809fcde215a255765ede0af3cef8687"} Jul 17 18:44:05.256771 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:05.256306 2572 scope.go:117] "RemoveContainer" containerID="66d65dfb764b00900931da505b778feada647bafaf3bf3a604a3523bd6520b55" Jul 17 18:44:05.256771 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:05.256586 2572 scope.go:117] "RemoveContainer" containerID="c83e1f010dc15dfc0a9434af4ac125af2809fcde215a255765ede0af3cef8687" Jul 17 18:44:05.256880 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:44:05.256820 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg_odh-ai-gateway-infra(4829e7a8-c318-4e87-90a0-898702853415)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg" podUID="4829e7a8-c318-4e87-90a0-898702853415" Jul 17 18:44:06.281036 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:06.280988 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg"] Jul 17 18:44:06.404496 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:06.404470 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg" Jul 17 18:44:06.549997 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:06.549911 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4829e7a8-c318-4e87-90a0-898702853415-maas-api-service-ca\") pod \"4829e7a8-c318-4e87-90a0-898702853415\" (UID: \"4829e7a8-c318-4e87-90a0-898702853415\") " Jul 17 18:44:06.550258 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:06.550235 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4829e7a8-c318-4e87-90a0-898702853415-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4829e7a8-c318-4e87-90a0-898702853415" (UID: "4829e7a8-c318-4e87-90a0-898702853415"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:44:06.650981 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:06.650953 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4829e7a8-c318-4e87-90a0-898702853415-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:44:07.267090 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:07.267049 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg" event={"ID":"4829e7a8-c318-4e87-90a0-898702853415","Type":"ContainerDied","Data":"bf3f61b6aa40ac415428ea8ba411856dfd927a8e5766156259cce3edede9bfd3"} Jul 17 18:44:07.267248 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:07.267116 2572 scope.go:117] "RemoveContainer" containerID="c83e1f010dc15dfc0a9434af4ac125af2809fcde215a255765ede0af3cef8687" Jul 17 18:44:07.267248 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:07.267061 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg" Jul 17 18:44:07.283130 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:07.283109 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg"] Jul 17 18:44:07.287149 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:07.287130 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-48sqg"] Jul 17 18:44:09.224587 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:09.224554 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4829e7a8-c318-4e87-90a0-898702853415" path="/var/lib/kubelet/pods/4829e7a8-c318-4e87-90a0-898702853415/volumes" Jul 17 18:44:11.008521 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.008488 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv"] Jul 17 18:44:11.008907 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.008804 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4829e7a8-c318-4e87-90a0-898702853415" containerName="revoke-keys" Jul 17 18:44:11.008907 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.008815 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4829e7a8-c318-4e87-90a0-898702853415" containerName="revoke-keys" Jul 17 18:44:11.008907 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.008826 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6648d623-a8f1-44bb-87d1-399f9bdf2d05" containerName="revoke-keys" Jul 17 18:44:11.008907 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.008831 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6648d623-a8f1-44bb-87d1-399f9bdf2d05" containerName="revoke-keys" Jul 17 18:44:11.008907 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.008838 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4829e7a8-c318-4e87-90a0-898702853415" containerName="revoke-keys" Jul 17 18:44:11.008907 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.008844 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4829e7a8-c318-4e87-90a0-898702853415" containerName="revoke-keys" Jul 17 18:44:11.008907 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.008856 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4829e7a8-c318-4e87-90a0-898702853415" containerName="revoke-keys" Jul 17 18:44:11.008907 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.008861 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4829e7a8-c318-4e87-90a0-898702853415" containerName="revoke-keys" Jul 17 18:44:11.008907 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.008905 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4829e7a8-c318-4e87-90a0-898702853415" containerName="revoke-keys" Jul 17 18:44:11.009221 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.008914 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6648d623-a8f1-44bb-87d1-399f9bdf2d05" containerName="revoke-keys" Jul 17 18:44:11.009221 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.008922 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4829e7a8-c318-4e87-90a0-898702853415" containerName="revoke-keys" Jul 17 18:44:11.013173 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.013156 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv" Jul 17 18:44:11.015570 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.015548 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:44:11.015669 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.015584 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:44:11.018955 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.018932 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv"] Jul 17 18:44:11.190748 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.190715 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e1903b-78ae-49f4-becc-d4987914053c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv\" (UID: \"31e1903b-78ae-49f4-becc-d4987914053c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv" Jul 17 18:44:11.291442 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.291414 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e1903b-78ae-49f4-becc-d4987914053c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv\" (UID: \"31e1903b-78ae-49f4-becc-d4987914053c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv" Jul 17 18:44:11.291996 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.291976 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e1903b-78ae-49f4-becc-d4987914053c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv\" (UID: \"31e1903b-78ae-49f4-becc-d4987914053c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv" Jul 17 18:44:11.324054 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.324003 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv" Jul 17 18:44:11.458791 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:11.458759 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv"] Jul 17 18:44:11.461052 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:44:11.461004 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e1903b_78ae_49f4_becc_d4987914053c.slice/crio-99e399b06d37e46ad51525a5d67dc2a2ad8858cc45498272625057172e436524 WatchSource:0}: Error finding container 99e399b06d37e46ad51525a5d67dc2a2ad8858cc45498272625057172e436524: Status 404 returned error can't find the container with id 99e399b06d37e46ad51525a5d67dc2a2ad8858cc45498272625057172e436524 Jul 17 18:44:12.294709 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:12.294627 2572 generic.go:356] "Generic (PLEG): container finished" podID="31e1903b-78ae-49f4-becc-d4987914053c" containerID="377ceb6c0090986fc5efa5d80f72c75258f3d076bdf82a18dfe40650ff6a39ba" exitCode=6 Jul 17 18:44:12.295068 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:12.294714 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv" event={"ID":"31e1903b-78ae-49f4-becc-d4987914053c","Type":"ContainerDied","Data":"377ceb6c0090986fc5efa5d80f72c75258f3d076bdf82a18dfe40650ff6a39ba"} Jul 17 18:44:12.295068 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:12.294757 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv" event={"ID":"31e1903b-78ae-49f4-becc-d4987914053c","Type":"ContainerStarted","Data":"99e399b06d37e46ad51525a5d67dc2a2ad8858cc45498272625057172e436524"} Jul 17 18:44:12.295068 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:12.294901 2572 scope.go:117] "RemoveContainer" containerID="377ceb6c0090986fc5efa5d80f72c75258f3d076bdf82a18dfe40650ff6a39ba" Jul 17 18:44:13.300971 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:13.300942 2572 generic.go:356] "Generic (PLEG): container finished" podID="31e1903b-78ae-49f4-becc-d4987914053c" containerID="4678ff920c24246f77ebda145be6ad8abfd156e5493f8b283e196ee737043071" exitCode=6 Jul 17 18:44:13.300971 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:13.300981 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv" event={"ID":"31e1903b-78ae-49f4-becc-d4987914053c","Type":"ContainerDied","Data":"4678ff920c24246f77ebda145be6ad8abfd156e5493f8b283e196ee737043071"} Jul 17 18:44:13.301493 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:13.301058 2572 scope.go:117] "RemoveContainer" containerID="377ceb6c0090986fc5efa5d80f72c75258f3d076bdf82a18dfe40650ff6a39ba" Jul 17 18:44:13.301493 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:13.301215 2572 scope.go:117] "RemoveContainer" containerID="4678ff920c24246f77ebda145be6ad8abfd156e5493f8b283e196ee737043071" Jul 17 18:44:13.301493 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:44:13.301423 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv_odh-ai-gateway-infra(31e1903b-78ae-49f4-becc-d4987914053c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv" podUID="31e1903b-78ae-49f4-becc-d4987914053c" Jul 17 18:44:14.308104 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:14.308077 2572 scope.go:117] "RemoveContainer" containerID="4678ff920c24246f77ebda145be6ad8abfd156e5493f8b283e196ee737043071" Jul 17 18:44:14.308479 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:44:14.308329 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv_odh-ai-gateway-infra(31e1903b-78ae-49f4-becc-d4987914053c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv" podUID="31e1903b-78ae-49f4-becc-d4987914053c" Jul 17 18:44:28.219388 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:28.219356 2572 scope.go:117] "RemoveContainer" containerID="4678ff920c24246f77ebda145be6ad8abfd156e5493f8b283e196ee737043071" Jul 17 18:44:29.372212 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:29.372182 2572 generic.go:356] "Generic (PLEG): container finished" podID="31e1903b-78ae-49f4-becc-d4987914053c" containerID="f4dfdb215f3c908524859f08100c96f019b044141ca98fb738e81c2fd831272c" exitCode=6 Jul 17 18:44:29.372212 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:29.372218 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv" event={"ID":"31e1903b-78ae-49f4-becc-d4987914053c","Type":"ContainerDied","Data":"f4dfdb215f3c908524859f08100c96f019b044141ca98fb738e81c2fd831272c"} Jul 17 18:44:29.372766 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:29.372265 2572 scope.go:117] "RemoveContainer" containerID="4678ff920c24246f77ebda145be6ad8abfd156e5493f8b283e196ee737043071" Jul 17 18:44:29.372766 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:29.372590 2572 scope.go:117] "RemoveContainer" containerID="f4dfdb215f3c908524859f08100c96f019b044141ca98fb738e81c2fd831272c" Jul 17 18:44:29.372881 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:44:29.372851 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv_odh-ai-gateway-infra(31e1903b-78ae-49f4-becc-d4987914053c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv" podUID="31e1903b-78ae-49f4-becc-d4987914053c" Jul 17 18:44:30.400135 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:30.400102 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv"] Jul 17 18:44:30.522228 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:30.522206 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv" Jul 17 18:44:30.545877 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:30.545852 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e1903b-78ae-49f4-becc-d4987914053c-maas-api-service-ca\") pod \"31e1903b-78ae-49f4-becc-d4987914053c\" (UID: \"31e1903b-78ae-49f4-becc-d4987914053c\") " Jul 17 18:44:30.546200 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:30.546178 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e1903b-78ae-49f4-becc-d4987914053c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "31e1903b-78ae-49f4-becc-d4987914053c" (UID: "31e1903b-78ae-49f4-becc-d4987914053c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:44:30.647082 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:30.647049 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31e1903b-78ae-49f4-becc-d4987914053c-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:44:31.383998 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:31.383965 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv" event={"ID":"31e1903b-78ae-49f4-becc-d4987914053c","Type":"ContainerDied","Data":"99e399b06d37e46ad51525a5d67dc2a2ad8858cc45498272625057172e436524"} Jul 17 18:44:31.384202 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:31.384040 2572 scope.go:117] "RemoveContainer" containerID="f4dfdb215f3c908524859f08100c96f019b044141ca98fb738e81c2fd831272c" Jul 17 18:44:31.384202 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:31.383974 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv" Jul 17 18:44:31.400048 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:31.400023 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv"] Jul 17 18:44:31.403655 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:31.403630 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-l7vfv"] Jul 17 18:44:33.225137 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:33.225103 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31e1903b-78ae-49f4-becc-d4987914053c" path="/var/lib/kubelet/pods/31e1903b-78ae-49f4-becc-d4987914053c/volumes" Jul 17 18:44:45.438577 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:45.438497 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc"] Jul 17 18:44:45.439087 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:45.438842 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31e1903b-78ae-49f4-becc-d4987914053c" containerName="revoke-keys" Jul 17 18:44:45.439087 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:45.438856 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="31e1903b-78ae-49f4-becc-d4987914053c" containerName="revoke-keys" Jul 17 18:44:45.439087 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:45.438878 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31e1903b-78ae-49f4-becc-d4987914053c" containerName="revoke-keys" Jul 17 18:44:45.439087 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:45.438883 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="31e1903b-78ae-49f4-becc-d4987914053c" containerName="revoke-keys" Jul 17 18:44:45.439087 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:45.438937 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="31e1903b-78ae-49f4-becc-d4987914053c" containerName="revoke-keys" Jul 17 18:44:45.439087 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:45.438945 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="31e1903b-78ae-49f4-becc-d4987914053c" containerName="revoke-keys" Jul 17 18:44:45.439087 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:45.438951 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="31e1903b-78ae-49f4-becc-d4987914053c" containerName="revoke-keys" Jul 17 18:44:45.439087 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:45.438959 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4829e7a8-c318-4e87-90a0-898702853415" containerName="revoke-keys" Jul 17 18:44:45.443374 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:45.443353 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc" Jul 17 18:44:45.445856 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:45.445837 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:44:45.445974 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:45.445842 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:44:45.450125 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:45.450102 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc"] Jul 17 18:44:45.584903 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:45.584867 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f209f555-739d-47df-9996-90e1afade4c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc\" (UID: \"f209f555-739d-47df-9996-90e1afade4c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc" Jul 17 18:44:45.685693 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:45.685656 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f209f555-739d-47df-9996-90e1afade4c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc\" (UID: \"f209f555-739d-47df-9996-90e1afade4c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc" Jul 17 18:44:45.686333 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:45.686313 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f209f555-739d-47df-9996-90e1afade4c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc\" (UID: \"f209f555-739d-47df-9996-90e1afade4c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc" Jul 17 18:44:45.755693 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:45.755596 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc" Jul 17 18:44:45.884038 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:45.883997 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc"] Jul 17 18:44:45.885670 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:44:45.885645 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf209f555_739d_47df_9996_90e1afade4c1.slice/crio-5e78a1356e12b869a2778f0ab2f542a37cee45bba743ca9e329fef3c6ba9936b WatchSource:0}: Error finding container 5e78a1356e12b869a2778f0ab2f542a37cee45bba743ca9e329fef3c6ba9936b: Status 404 returned error can't find the container with id 5e78a1356e12b869a2778f0ab2f542a37cee45bba743ca9e329fef3c6ba9936b Jul 17 18:44:46.447485 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:46.447407 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc" event={"ID":"f209f555-739d-47df-9996-90e1afade4c1","Type":"ContainerStarted","Data":"7a2365bc67fe7156f4fd25c4207a9b1082c89ef7b59d5687165f94ca28b49319"} Jul 17 18:44:46.447485 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:46.447455 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc" event={"ID":"f209f555-739d-47df-9996-90e1afade4c1","Type":"ContainerStarted","Data":"5e78a1356e12b869a2778f0ab2f542a37cee45bba743ca9e329fef3c6ba9936b"} Jul 17 18:44:46.463958 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:46.463909 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc" podStartSLOduration=1.463895617 podStartE2EDuration="1.463895617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:46.462429845 +0000 UTC m=+2881.871857452" watchObservedRunningTime="2026-07-17 18:44:46.463895617 +0000 UTC m=+2881.873323222" Jul 17 18:44:47.452895 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:47.452864 2572 generic.go:356] "Generic (PLEG): container finished" podID="f209f555-739d-47df-9996-90e1afade4c1" containerID="7a2365bc67fe7156f4fd25c4207a9b1082c89ef7b59d5687165f94ca28b49319" exitCode=6 Jul 17 18:44:47.453375 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:47.452938 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc" event={"ID":"f209f555-739d-47df-9996-90e1afade4c1","Type":"ContainerDied","Data":"7a2365bc67fe7156f4fd25c4207a9b1082c89ef7b59d5687165f94ca28b49319"} Jul 17 18:44:47.453375 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:47.453166 2572 scope.go:117] "RemoveContainer" containerID="7a2365bc67fe7156f4fd25c4207a9b1082c89ef7b59d5687165f94ca28b49319" Jul 17 18:44:48.458803 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:48.458715 2572 generic.go:356] "Generic (PLEG): container finished" podID="f209f555-739d-47df-9996-90e1afade4c1" containerID="82936e52117b7794cce3b9aaa823cf93d84d275f36575aaf1bd019bd009844cd" exitCode=6 Jul 17 18:44:48.459189 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:48.458806 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc" event={"ID":"f209f555-739d-47df-9996-90e1afade4c1","Type":"ContainerDied","Data":"82936e52117b7794cce3b9aaa823cf93d84d275f36575aaf1bd019bd009844cd"} Jul 17 18:44:48.459189 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:48.458890 2572 scope.go:117] "RemoveContainer" containerID="7a2365bc67fe7156f4fd25c4207a9b1082c89ef7b59d5687165f94ca28b49319" Jul 17 18:44:48.459189 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:48.459133 2572 scope.go:117] "RemoveContainer" containerID="82936e52117b7794cce3b9aaa823cf93d84d275f36575aaf1bd019bd009844cd" Jul 17 18:44:48.459375 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:44:48.459352 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc_odh-ai-gateway-infra(f209f555-739d-47df-9996-90e1afade4c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc" podUID="f209f555-739d-47df-9996-90e1afade4c1" Jul 17 18:44:49.464801 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:49.464775 2572 scope.go:117] "RemoveContainer" containerID="82936e52117b7794cce3b9aaa823cf93d84d275f36575aaf1bd019bd009844cd" Jul 17 18:44:49.465224 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:44:49.464938 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc_odh-ai-gateway-infra(f209f555-739d-47df-9996-90e1afade4c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc" podUID="f209f555-739d-47df-9996-90e1afade4c1" Jul 17 18:44:57.574560 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:57.574526 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn"] Jul 17 18:44:57.574991 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:57.574862 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31e1903b-78ae-49f4-becc-d4987914053c" containerName="revoke-keys" Jul 17 18:44:57.574991 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:57.574875 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="31e1903b-78ae-49f4-becc-d4987914053c" containerName="revoke-keys" Jul 17 18:44:57.577742 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:57.577728 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn" Jul 17 18:44:57.585782 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:57.585755 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn"] Jul 17 18:44:57.688926 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:57.688896 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f81d496a-d9ee-4b8c-8275-93cbecf51a8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn\" (UID: \"f81d496a-d9ee-4b8c-8275-93cbecf51a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn" Jul 17 18:44:57.789405 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:57.789376 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f81d496a-d9ee-4b8c-8275-93cbecf51a8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn\" (UID: \"f81d496a-d9ee-4b8c-8275-93cbecf51a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn" Jul 17 18:44:57.790000 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:57.789979 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f81d496a-d9ee-4b8c-8275-93cbecf51a8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn\" (UID: \"f81d496a-d9ee-4b8c-8275-93cbecf51a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn" Jul 17 18:44:57.888517 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:57.888453 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn" Jul 17 18:44:58.038319 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:58.038295 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn"] Jul 17 18:44:58.040143 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:44:58.040114 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81d496a_d9ee_4b8c_8275_93cbecf51a8e.slice/crio-087600b7db1451ce00075449f3fa55469d0e10b87559cd271c76e6d1aa6f5abf WatchSource:0}: Error finding container 087600b7db1451ce00075449f3fa55469d0e10b87559cd271c76e6d1aa6f5abf: Status 404 returned error can't find the container with id 087600b7db1451ce00075449f3fa55469d0e10b87559cd271c76e6d1aa6f5abf Jul 17 18:44:58.503201 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:58.503099 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn" event={"ID":"f81d496a-d9ee-4b8c-8275-93cbecf51a8e","Type":"ContainerStarted","Data":"a52dd76778685def3466140b2dcd3f2699d78140343698ab6ebdd7d47c80d533"} Jul 17 18:44:58.503201 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:58.503151 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn" event={"ID":"f81d496a-d9ee-4b8c-8275-93cbecf51a8e","Type":"ContainerStarted","Data":"087600b7db1451ce00075449f3fa55469d0e10b87559cd271c76e6d1aa6f5abf"} Jul 17 18:44:58.522487 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:58.522435 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn" podStartSLOduration=1.522420494 podStartE2EDuration="1.522420494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:58.520603883 +0000 UTC m=+2893.930031516" watchObservedRunningTime="2026-07-17 18:44:58.522420494 +0000 UTC m=+2893.931848103" Jul 17 18:44:59.508544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:59.508509 2572 generic.go:356] "Generic (PLEG): container finished" podID="f81d496a-d9ee-4b8c-8275-93cbecf51a8e" containerID="a52dd76778685def3466140b2dcd3f2699d78140343698ab6ebdd7d47c80d533" exitCode=6 Jul 17 18:44:59.508923 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:59.508575 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn" event={"ID":"f81d496a-d9ee-4b8c-8275-93cbecf51a8e","Type":"ContainerDied","Data":"a52dd76778685def3466140b2dcd3f2699d78140343698ab6ebdd7d47c80d533"} Jul 17 18:44:59.508923 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:44:59.508797 2572 scope.go:117] "RemoveContainer" containerID="a52dd76778685def3466140b2dcd3f2699d78140343698ab6ebdd7d47c80d533" Jul 17 18:45:00.136277 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:00.136250 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738565-rdx25"] Jul 17 18:45:00.139299 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:00.139278 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738565-rdx25" Jul 17 18:45:00.139494 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:00.139476 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9de544-29738565-t76pw"] Jul 17 18:45:00.142309 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:00.142289 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9de544-29738565-t76pw" Jul 17 18:45:00.147205 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:00.147184 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738565-rdx25"] Jul 17 18:45:00.157026 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:00.155107 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9de544-29738565-t76pw"] Jul 17 18:45:00.185298 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:00.185276 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738565-rdx25" Jul 17 18:45:00.191208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:00.191187 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9de544-29738565-t76pw" Jul 17 18:45:00.318323 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:00.318296 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738565-rdx25"] Jul 17 18:45:00.319920 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:45:00.319890 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301834ae_965c_4282_b448_735e2b2aa5cb.slice/crio-cc7a34b69ef6aa9705b6ec88ca3d250601b4153663df745815ffa874b8892aab WatchSource:0}: Error finding container cc7a34b69ef6aa9705b6ec88ca3d250601b4153663df745815ffa874b8892aab: Status 404 returned error can't find the container with id cc7a34b69ef6aa9705b6ec88ca3d250601b4153663df745815ffa874b8892aab Jul 17 18:45:00.336991 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:00.336968 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9de544-29738565-t76pw"] Jul 17 18:45:00.340396 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:45:00.340323 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1800f7_6b69_4fd1_bd6c_8e96b0cf3551.slice/crio-ae6417b461cb98801b6181b8e70e657b76db234d7f192f6a2325657d7d2c7958 WatchSource:0}: Error finding container ae6417b461cb98801b6181b8e70e657b76db234d7f192f6a2325657d7d2c7958: Status 404 returned error can't find the container with id ae6417b461cb98801b6181b8e70e657b76db234d7f192f6a2325657d7d2c7958 Jul 17 18:45:00.514682 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:00.514643 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9de544-29738565-t76pw" event={"ID":"4c1800f7-6b69-4fd1-bd6c-8e96b0cf3551","Type":"ContainerStarted","Data":"ae6417b461cb98801b6181b8e70e657b76db234d7f192f6a2325657d7d2c7958"} Jul 17 18:45:00.515840 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:00.515817 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738565-rdx25" event={"ID":"301834ae-965c-4282-b448-735e2b2aa5cb","Type":"ContainerStarted","Data":"cc7a34b69ef6aa9705b6ec88ca3d250601b4153663df745815ffa874b8892aab"} Jul 17 18:45:00.517448 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:00.517428 2572 generic.go:356] "Generic (PLEG): container finished" podID="f81d496a-d9ee-4b8c-8275-93cbecf51a8e" containerID="52b81df93733d472c6bed0b10c55116ae8242076dd96758bb4b3ab15eec56dd6" exitCode=6 Jul 17 18:45:00.517561 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:00.517512 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn" event={"ID":"f81d496a-d9ee-4b8c-8275-93cbecf51a8e","Type":"ContainerDied","Data":"52b81df93733d472c6bed0b10c55116ae8242076dd96758bb4b3ab15eec56dd6"} Jul 17 18:45:00.517605 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:00.517587 2572 scope.go:117] "RemoveContainer" containerID="a52dd76778685def3466140b2dcd3f2699d78140343698ab6ebdd7d47c80d533" Jul 17 18:45:00.517802 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:00.517786 2572 scope.go:117] "RemoveContainer" containerID="52b81df93733d472c6bed0b10c55116ae8242076dd96758bb4b3ab15eec56dd6" Jul 17 18:45:00.517996 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:45:00.517980 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn_odh-ai-gateway-infra(f81d496a-d9ee-4b8c-8275-93cbecf51a8e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn" podUID="f81d496a-d9ee-4b8c-8275-93cbecf51a8e" Jul 17 18:45:01.523863 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:01.523765 2572 generic.go:356] "Generic (PLEG): container finished" podID="4c1800f7-6b69-4fd1-bd6c-8e96b0cf3551" containerID="e81e48b1d4be65cae68c51d11bca22cc1d78a84f4f3b31e275df8c6389226eb6" exitCode=0 Jul 17 18:45:01.524326 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:01.523865 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9de544-29738565-t76pw" event={"ID":"4c1800f7-6b69-4fd1-bd6c-8e96b0cf3551","Type":"ContainerDied","Data":"e81e48b1d4be65cae68c51d11bca22cc1d78a84f4f3b31e275df8c6389226eb6"} Jul 17 18:45:01.525400 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:01.525371 2572 generic.go:356] "Generic (PLEG): container finished" podID="301834ae-965c-4282-b448-735e2b2aa5cb" containerID="8a74969aa9da58bbd0d67043a87294f82bb73fed3e0e1aea8fb1bd02c21030cf" exitCode=0 Jul 17 18:45:01.525551 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:01.525470 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738565-rdx25" event={"ID":"301834ae-965c-4282-b448-735e2b2aa5cb","Type":"ContainerDied","Data":"8a74969aa9da58bbd0d67043a87294f82bb73fed3e0e1aea8fb1bd02c21030cf"} Jul 17 18:45:01.527094 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:01.527072 2572 scope.go:117] "RemoveContainer" containerID="52b81df93733d472c6bed0b10c55116ae8242076dd96758bb4b3ab15eec56dd6" Jul 17 18:45:01.527285 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:45:01.527267 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn_odh-ai-gateway-infra(f81d496a-d9ee-4b8c-8275-93cbecf51a8e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn" podUID="f81d496a-d9ee-4b8c-8275-93cbecf51a8e" Jul 17 18:45:02.663133 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:02.663106 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738565-rdx25" Jul 17 18:45:02.690772 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:02.690733 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9de544-29738565-t76pw" Jul 17 18:45:03.537731 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.537698 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9de544-29738565-t76pw" Jul 17 18:45:03.537731 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.537705 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9de544-29738565-t76pw" event={"ID":"4c1800f7-6b69-4fd1-bd6c-8e96b0cf3551","Type":"ContainerDied","Data":"ae6417b461cb98801b6181b8e70e657b76db234d7f192f6a2325657d7d2c7958"} Jul 17 18:45:03.537959 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.537757 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae6417b461cb98801b6181b8e70e657b76db234d7f192f6a2325657d7d2c7958" Jul 17 18:45:03.539039 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.539000 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738565-rdx25" Jul 17 18:45:03.539039 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.539027 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738565-rdx25" event={"ID":"301834ae-965c-4282-b448-735e2b2aa5cb","Type":"ContainerDied","Data":"cc7a34b69ef6aa9705b6ec88ca3d250601b4153663df745815ffa874b8892aab"} Jul 17 18:45:03.539208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.539070 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc7a34b69ef6aa9705b6ec88ca3d250601b4153663df745815ffa874b8892aab" Jul 17 18:45:03.602776 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.602746 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw"] Jul 17 18:45:03.603062 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.603051 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="301834ae-965c-4282-b448-735e2b2aa5cb" containerName="cleanup" Jul 17 18:45:03.603118 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.603063 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="301834ae-965c-4282-b448-735e2b2aa5cb" containerName="cleanup" Jul 17 18:45:03.603118 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.603072 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c1800f7-6b69-4fd1-bd6c-8e96b0cf3551" containerName="cleanup" Jul 17 18:45:03.603118 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.603078 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c1800f7-6b69-4fd1-bd6c-8e96b0cf3551" containerName="cleanup" Jul 17 18:45:03.603214 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.603121 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c1800f7-6b69-4fd1-bd6c-8e96b0cf3551" containerName="cleanup" Jul 17 18:45:03.603214 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.603130 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="301834ae-965c-4282-b448-735e2b2aa5cb" containerName="cleanup" Jul 17 18:45:03.605896 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.605882 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw" Jul 17 18:45:03.613028 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.612982 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw"] Jul 17 18:45:03.636288 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.636254 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94826487-b1dd-4242-8600-b82d74095fb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw\" (UID: \"94826487-b1dd-4242-8600-b82d74095fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw" Jul 17 18:45:03.737346 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.737319 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94826487-b1dd-4242-8600-b82d74095fb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw\" (UID: \"94826487-b1dd-4242-8600-b82d74095fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw" Jul 17 18:45:03.737942 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.737921 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94826487-b1dd-4242-8600-b82d74095fb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw\" (UID: \"94826487-b1dd-4242-8600-b82d74095fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw" Jul 17 18:45:03.917077 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:03.916961 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw" Jul 17 18:45:04.041775 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:04.041744 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw"] Jul 17 18:45:04.043488 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:45:04.043463 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94826487_b1dd_4242_8600_b82d74095fb8.slice/crio-70189d8daa164f162305695fa665e2ddd8b02805eaabc265b81ad764b5ac0b64 WatchSource:0}: Error finding container 70189d8daa164f162305695fa665e2ddd8b02805eaabc265b81ad764b5ac0b64: Status 404 returned error can't find the container with id 70189d8daa164f162305695fa665e2ddd8b02805eaabc265b81ad764b5ac0b64 Jul 17 18:45:04.220220 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:04.220150 2572 scope.go:117] "RemoveContainer" containerID="82936e52117b7794cce3b9aaa823cf93d84d275f36575aaf1bd019bd009844cd" Jul 17 18:45:04.547059 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:04.547031 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw" event={"ID":"94826487-b1dd-4242-8600-b82d74095fb8","Type":"ContainerStarted","Data":"d0bc48a4935c5ca68e4a927e10edaa91bb6fbd0eb89f229246bed25a53000719"} Jul 17 18:45:04.547182 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:04.547086 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw" event={"ID":"94826487-b1dd-4242-8600-b82d74095fb8","Type":"ContainerStarted","Data":"70189d8daa164f162305695fa665e2ddd8b02805eaabc265b81ad764b5ac0b64"} Jul 17 18:45:04.563090 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:04.563046 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw" podStartSLOduration=1.563030878 podStartE2EDuration="1.563030878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:04.561610098 +0000 UTC m=+2899.971037704" watchObservedRunningTime="2026-07-17 18:45:04.563030878 +0000 UTC m=+2899.972458482" Jul 17 18:45:05.553073 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:05.553042 2572 generic.go:356] "Generic (PLEG): container finished" podID="f209f555-739d-47df-9996-90e1afade4c1" containerID="bb448ae4683a0ac01d6881c92de02f4ecf5c20fb3035ed633d05777284224d5c" exitCode=6 Jul 17 18:45:05.553501 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:05.553121 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc" event={"ID":"f209f555-739d-47df-9996-90e1afade4c1","Type":"ContainerDied","Data":"bb448ae4683a0ac01d6881c92de02f4ecf5c20fb3035ed633d05777284224d5c"} Jul 17 18:45:05.553501 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:05.553191 2572 scope.go:117] "RemoveContainer" containerID="82936e52117b7794cce3b9aaa823cf93d84d275f36575aaf1bd019bd009844cd" Jul 17 18:45:05.553501 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:05.553452 2572 scope.go:117] "RemoveContainer" containerID="bb448ae4683a0ac01d6881c92de02f4ecf5c20fb3035ed633d05777284224d5c" Jul 17 18:45:05.553710 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:45:05.553690 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc_odh-ai-gateway-infra(f209f555-739d-47df-9996-90e1afade4c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc" podUID="f209f555-739d-47df-9996-90e1afade4c1" Jul 17 18:45:05.554718 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:05.554687 2572 generic.go:356] "Generic (PLEG): container finished" podID="94826487-b1dd-4242-8600-b82d74095fb8" containerID="d0bc48a4935c5ca68e4a927e10edaa91bb6fbd0eb89f229246bed25a53000719" exitCode=6 Jul 17 18:45:05.554800 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:05.554740 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw" event={"ID":"94826487-b1dd-4242-8600-b82d74095fb8","Type":"ContainerDied","Data":"d0bc48a4935c5ca68e4a927e10edaa91bb6fbd0eb89f229246bed25a53000719"} Jul 17 18:45:05.554939 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:05.554927 2572 scope.go:117] "RemoveContainer" containerID="d0bc48a4935c5ca68e4a927e10edaa91bb6fbd0eb89f229246bed25a53000719" Jul 17 18:45:06.561601 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:06.561568 2572 generic.go:356] "Generic (PLEG): container finished" podID="94826487-b1dd-4242-8600-b82d74095fb8" containerID="37ad4ac7739cbb52c70fcc4463e2b61781ab00cca219781df65088e0f3317948" exitCode=6 Jul 17 18:45:06.561993 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:06.561616 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw" event={"ID":"94826487-b1dd-4242-8600-b82d74095fb8","Type":"ContainerDied","Data":"37ad4ac7739cbb52c70fcc4463e2b61781ab00cca219781df65088e0f3317948"} Jul 17 18:45:06.561993 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:06.561676 2572 scope.go:117] "RemoveContainer" containerID="d0bc48a4935c5ca68e4a927e10edaa91bb6fbd0eb89f229246bed25a53000719" Jul 17 18:45:06.561993 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:06.561866 2572 scope.go:117] "RemoveContainer" containerID="37ad4ac7739cbb52c70fcc4463e2b61781ab00cca219781df65088e0f3317948" Jul 17 18:45:06.562156 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:45:06.562072 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw_odh-ai-gateway-infra(94826487-b1dd-4242-8600-b82d74095fb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw" podUID="94826487-b1dd-4242-8600-b82d74095fb8" Jul 17 18:45:06.579438 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:06.579411 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc"] Jul 17 18:45:06.696970 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:06.696948 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc" Jul 17 18:45:06.761495 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:06.761455 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f209f555-739d-47df-9996-90e1afade4c1-maas-api-service-ca\") pod \"f209f555-739d-47df-9996-90e1afade4c1\" (UID: \"f209f555-739d-47df-9996-90e1afade4c1\") " Jul 17 18:45:06.761800 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:06.761779 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f209f555-739d-47df-9996-90e1afade4c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f209f555-739d-47df-9996-90e1afade4c1" (UID: "f209f555-739d-47df-9996-90e1afade4c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:45:06.862353 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:06.862280 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f209f555-739d-47df-9996-90e1afade4c1-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:45:07.567079 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:07.567046 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc" Jul 17 18:45:07.567079 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:07.567055 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc" event={"ID":"f209f555-739d-47df-9996-90e1afade4c1","Type":"ContainerDied","Data":"5e78a1356e12b869a2778f0ab2f542a37cee45bba743ca9e329fef3c6ba9936b"} Jul 17 18:45:07.567566 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:07.567116 2572 scope.go:117] "RemoveContainer" containerID="bb448ae4683a0ac01d6881c92de02f4ecf5c20fb3035ed633d05777284224d5c" Jul 17 18:45:07.568943 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:07.568908 2572 scope.go:117] "RemoveContainer" containerID="37ad4ac7739cbb52c70fcc4463e2b61781ab00cca219781df65088e0f3317948" Jul 17 18:45:07.569302 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:45:07.569221 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw_odh-ai-gateway-infra(94826487-b1dd-4242-8600-b82d74095fb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw" podUID="94826487-b1dd-4242-8600-b82d74095fb8" Jul 17 18:45:07.599343 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:07.599319 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc"] Jul 17 18:45:07.603157 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:07.603136 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cwcdc"] Jul 17 18:45:09.224916 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:09.224871 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f209f555-739d-47df-9996-90e1afade4c1" path="/var/lib/kubelet/pods/f209f555-739d-47df-9996-90e1afade4c1/volumes" Jul 17 18:45:11.055155 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:11.055119 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf"] Jul 17 18:45:11.055520 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:11.055443 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f209f555-739d-47df-9996-90e1afade4c1" containerName="revoke-keys" Jul 17 18:45:11.055520 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:11.055454 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f209f555-739d-47df-9996-90e1afade4c1" containerName="revoke-keys" Jul 17 18:45:11.055520 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:11.055465 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f209f555-739d-47df-9996-90e1afade4c1" containerName="revoke-keys" Jul 17 18:45:11.055520 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:11.055471 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f209f555-739d-47df-9996-90e1afade4c1" containerName="revoke-keys" Jul 17 18:45:11.055634 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:11.055529 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f209f555-739d-47df-9996-90e1afade4c1" containerName="revoke-keys" Jul 17 18:45:11.055634 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:11.055537 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f209f555-739d-47df-9996-90e1afade4c1" containerName="revoke-keys" Jul 17 18:45:11.059825 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:11.059809 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf" Jul 17 18:45:11.064803 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:11.064776 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf"] Jul 17 18:45:11.090501 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:11.090473 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf\" (UID: \"47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf" Jul 17 18:45:11.191096 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:11.191063 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf\" (UID: \"47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf" Jul 17 18:45:11.191656 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:11.191637 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf\" (UID: \"47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf" Jul 17 18:45:11.371472 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:11.371389 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf" Jul 17 18:45:11.496650 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:11.496623 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf"] Jul 17 18:45:11.498403 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:45:11.498376 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ca5c3d_3d3f_4811_bd2a_b13ea9ce8546.slice/crio-e135f297dce3e19dffd1a667a09f4f74e72a4238f754c1c726e0a149c1a21db5 WatchSource:0}: Error finding container e135f297dce3e19dffd1a667a09f4f74e72a4238f754c1c726e0a149c1a21db5: Status 404 returned error can't find the container with id e135f297dce3e19dffd1a667a09f4f74e72a4238f754c1c726e0a149c1a21db5 Jul 17 18:45:11.587178 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:11.587153 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf" event={"ID":"47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546","Type":"ContainerStarted","Data":"e135f297dce3e19dffd1a667a09f4f74e72a4238f754c1c726e0a149c1a21db5"} Jul 17 18:45:12.593475 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:12.593439 2572 generic.go:356] "Generic (PLEG): container finished" podID="47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546" containerID="414b4c3b56c07084a1ea9433de0411505fc642550cfb00cefdbb55ad34ea1e16" exitCode=6 Jul 17 18:45:12.593918 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:12.593546 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf" event={"ID":"47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546","Type":"ContainerDied","Data":"414b4c3b56c07084a1ea9433de0411505fc642550cfb00cefdbb55ad34ea1e16"} Jul 17 18:45:12.593918 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:12.593772 2572 scope.go:117] "RemoveContainer" containerID="414b4c3b56c07084a1ea9433de0411505fc642550cfb00cefdbb55ad34ea1e16" Jul 17 18:45:13.599764 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:13.599737 2572 generic.go:356] "Generic (PLEG): container finished" podID="47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546" containerID="39f97e7890ebc61e7fab1bd471f440a0dbf2775671497ac7512c0c2d447f6862" exitCode=6 Jul 17 18:45:13.600232 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:13.599805 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf" event={"ID":"47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546","Type":"ContainerDied","Data":"39f97e7890ebc61e7fab1bd471f440a0dbf2775671497ac7512c0c2d447f6862"} Jul 17 18:45:13.600232 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:13.599878 2572 scope.go:117] "RemoveContainer" containerID="414b4c3b56c07084a1ea9433de0411505fc642550cfb00cefdbb55ad34ea1e16" Jul 17 18:45:13.600232 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:13.600035 2572 scope.go:117] "RemoveContainer" containerID="39f97e7890ebc61e7fab1bd471f440a0dbf2775671497ac7512c0c2d447f6862" Jul 17 18:45:13.600360 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:45:13.600271 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf_odh-ai-gateway-infra(47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf" podUID="47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546" Jul 17 18:45:14.606276 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:14.606245 2572 scope.go:117] "RemoveContainer" containerID="39f97e7890ebc61e7fab1bd471f440a0dbf2775671497ac7512c0c2d447f6862" Jul 17 18:45:14.606694 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:45:14.606412 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf_odh-ai-gateway-infra(47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf" podUID="47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546" Jul 17 18:45:17.219779 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:17.219752 2572 scope.go:117] "RemoveContainer" containerID="52b81df93733d472c6bed0b10c55116ae8242076dd96758bb4b3ab15eec56dd6" Jul 17 18:45:17.620095 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:17.620066 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn" event={"ID":"f81d496a-d9ee-4b8c-8275-93cbecf51a8e","Type":"ContainerStarted","Data":"13c0bc6ef39d9534d65dc12158963ad3de3cffccd0364e6214ff1d0e7fbd22a3"} Jul 17 18:45:18.626228 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:18.626198 2572 generic.go:356] "Generic (PLEG): container finished" podID="f81d496a-d9ee-4b8c-8275-93cbecf51a8e" containerID="13c0bc6ef39d9534d65dc12158963ad3de3cffccd0364e6214ff1d0e7fbd22a3" exitCode=6 Jul 17 18:45:18.626671 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:18.626271 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn" event={"ID":"f81d496a-d9ee-4b8c-8275-93cbecf51a8e","Type":"ContainerDied","Data":"13c0bc6ef39d9534d65dc12158963ad3de3cffccd0364e6214ff1d0e7fbd22a3"} Jul 17 18:45:18.626671 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:18.626330 2572 scope.go:117] "RemoveContainer" containerID="52b81df93733d472c6bed0b10c55116ae8242076dd96758bb4b3ab15eec56dd6" Jul 17 18:45:18.626671 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:18.626622 2572 scope.go:117] "RemoveContainer" containerID="13c0bc6ef39d9534d65dc12158963ad3de3cffccd0364e6214ff1d0e7fbd22a3" Jul 17 18:45:18.626889 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:45:18.626871 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn_odh-ai-gateway-infra(f81d496a-d9ee-4b8c-8275-93cbecf51a8e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn" podUID="f81d496a-d9ee-4b8c-8275-93cbecf51a8e" Jul 17 18:45:18.651313 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:18.651291 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn"] Jul 17 18:45:19.769732 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:19.769704 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn" Jul 17 18:45:19.851544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:19.851517 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f81d496a-d9ee-4b8c-8275-93cbecf51a8e-maas-api-service-ca\") pod \"f81d496a-d9ee-4b8c-8275-93cbecf51a8e\" (UID: \"f81d496a-d9ee-4b8c-8275-93cbecf51a8e\") " Jul 17 18:45:19.851853 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:19.851830 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81d496a-d9ee-4b8c-8275-93cbecf51a8e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f81d496a-d9ee-4b8c-8275-93cbecf51a8e" (UID: "f81d496a-d9ee-4b8c-8275-93cbecf51a8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:45:19.952828 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:19.952758 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f81d496a-d9ee-4b8c-8275-93cbecf51a8e-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:45:20.220170 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:20.220102 2572 scope.go:117] "RemoveContainer" containerID="37ad4ac7739cbb52c70fcc4463e2b61781ab00cca219781df65088e0f3317948" Jul 17 18:45:20.638595 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:20.638566 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn" Jul 17 18:45:20.638798 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:20.638595 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn" event={"ID":"f81d496a-d9ee-4b8c-8275-93cbecf51a8e","Type":"ContainerDied","Data":"087600b7db1451ce00075449f3fa55469d0e10b87559cd271c76e6d1aa6f5abf"} Jul 17 18:45:20.638798 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:20.638691 2572 scope.go:117] "RemoveContainer" containerID="13c0bc6ef39d9534d65dc12158963ad3de3cffccd0364e6214ff1d0e7fbd22a3" Jul 17 18:45:20.640585 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:20.640555 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw" event={"ID":"94826487-b1dd-4242-8600-b82d74095fb8","Type":"ContainerStarted","Data":"fe08fcde0d648678c89a580de420554aff613c0d3d933d9b7e3eab56d3ad26fb"} Jul 17 18:45:20.672516 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:20.672496 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn"] Jul 17 18:45:20.678245 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:20.678227 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-tf8nn"] Jul 17 18:45:21.224636 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:21.224600 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f81d496a-d9ee-4b8c-8275-93cbecf51a8e" path="/var/lib/kubelet/pods/f81d496a-d9ee-4b8c-8275-93cbecf51a8e/volumes" Jul 17 18:45:21.647487 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:21.647456 2572 generic.go:356] "Generic (PLEG): container finished" podID="94826487-b1dd-4242-8600-b82d74095fb8" containerID="fe08fcde0d648678c89a580de420554aff613c0d3d933d9b7e3eab56d3ad26fb" exitCode=6 Jul 17 18:45:21.647883 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:21.647538 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw" event={"ID":"94826487-b1dd-4242-8600-b82d74095fb8","Type":"ContainerDied","Data":"fe08fcde0d648678c89a580de420554aff613c0d3d933d9b7e3eab56d3ad26fb"} Jul 17 18:45:21.647883 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:21.647600 2572 scope.go:117] "RemoveContainer" containerID="37ad4ac7739cbb52c70fcc4463e2b61781ab00cca219781df65088e0f3317948" Jul 17 18:45:21.647883 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:21.647832 2572 scope.go:117] "RemoveContainer" containerID="fe08fcde0d648678c89a580de420554aff613c0d3d933d9b7e3eab56d3ad26fb" Jul 17 18:45:21.648080 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:45:21.648060 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw_odh-ai-gateway-infra(94826487-b1dd-4242-8600-b82d74095fb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw" podUID="94826487-b1dd-4242-8600-b82d74095fb8" Jul 17 18:45:21.673890 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:21.673867 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw"] Jul 17 18:45:22.786680 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:22.786657 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw" Jul 17 18:45:22.874284 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:22.874252 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94826487-b1dd-4242-8600-b82d74095fb8-maas-api-service-ca\") pod \"94826487-b1dd-4242-8600-b82d74095fb8\" (UID: \"94826487-b1dd-4242-8600-b82d74095fb8\") " Jul 17 18:45:22.874576 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:22.874556 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94826487-b1dd-4242-8600-b82d74095fb8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94826487-b1dd-4242-8600-b82d74095fb8" (UID: "94826487-b1dd-4242-8600-b82d74095fb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:45:22.975522 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:22.975456 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94826487-b1dd-4242-8600-b82d74095fb8-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:45:23.659338 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:23.659306 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw" Jul 17 18:45:23.659534 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:23.659337 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw" event={"ID":"94826487-b1dd-4242-8600-b82d74095fb8","Type":"ContainerDied","Data":"70189d8daa164f162305695fa665e2ddd8b02805eaabc265b81ad764b5ac0b64"} Jul 17 18:45:23.659534 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:23.659397 2572 scope.go:117] "RemoveContainer" containerID="fe08fcde0d648678c89a580de420554aff613c0d3d933d9b7e3eab56d3ad26fb" Jul 17 18:45:23.675576 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:23.675551 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw"] Jul 17 18:45:23.678228 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:23.678208 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-p4tsw"] Jul 17 18:45:25.224479 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:25.224448 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94826487-b1dd-4242-8600-b82d74095fb8" path="/var/lib/kubelet/pods/94826487-b1dd-4242-8600-b82d74095fb8/volumes" Jul 17 18:45:27.219880 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:27.219847 2572 scope.go:117] "RemoveContainer" containerID="39f97e7890ebc61e7fab1bd471f440a0dbf2775671497ac7512c0c2d447f6862" Jul 17 18:45:27.680136 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:27.680107 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf" event={"ID":"47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546","Type":"ContainerStarted","Data":"b76827260730270036c573e5a684add4a2b6247e2d9fb276e46ff8c9953b2c4b"} Jul 17 18:45:27.697224 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:27.697186 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf" podStartSLOduration=16.697172174 podStartE2EDuration="16.697172174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:27.695493348 +0000 UTC m=+2923.104920967" watchObservedRunningTime="2026-07-17 18:45:27.697172174 +0000 UTC m=+2923.106599813" Jul 17 18:45:28.686459 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:28.686426 2572 generic.go:356] "Generic (PLEG): container finished" podID="47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546" containerID="b76827260730270036c573e5a684add4a2b6247e2d9fb276e46ff8c9953b2c4b" exitCode=6 Jul 17 18:45:28.686890 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:28.686497 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf" event={"ID":"47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546","Type":"ContainerDied","Data":"b76827260730270036c573e5a684add4a2b6247e2d9fb276e46ff8c9953b2c4b"} Jul 17 18:45:28.686890 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:28.686558 2572 scope.go:117] "RemoveContainer" containerID="39f97e7890ebc61e7fab1bd471f440a0dbf2775671497ac7512c0c2d447f6862" Jul 17 18:45:28.686890 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:28.686806 2572 scope.go:117] "RemoveContainer" containerID="b76827260730270036c573e5a684add4a2b6247e2d9fb276e46ff8c9953b2c4b" Jul 17 18:45:28.687073 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:45:28.687053 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf_odh-ai-gateway-infra(47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf" podUID="47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546" Jul 17 18:45:28.706474 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:28.706450 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf"] Jul 17 18:45:29.817662 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:29.817640 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf" Jul 17 18:45:29.929816 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:29.929782 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546-maas-api-service-ca\") pod \"47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546\" (UID: \"47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546\") " Jul 17 18:45:29.930140 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:29.930118 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546" (UID: "47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:45:30.030898 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:30.030872 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:45:30.697782 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:30.697741 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf" event={"ID":"47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546","Type":"ContainerDied","Data":"e135f297dce3e19dffd1a667a09f4f74e72a4238f754c1c726e0a149c1a21db5"} Jul 17 18:45:30.697782 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:30.697778 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf" Jul 17 18:45:30.698053 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:30.697816 2572 scope.go:117] "RemoveContainer" containerID="b76827260730270036c573e5a684add4a2b6247e2d9fb276e46ff8c9953b2c4b" Jul 17 18:45:30.721615 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:30.721589 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf"] Jul 17 18:45:30.725415 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:30.725393 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kb4bf"] Jul 17 18:45:31.224302 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:31.224270 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546" path="/var/lib/kubelet/pods/47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546/volumes" Jul 17 18:45:45.485859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.485829 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf"] Jul 17 18:45:45.486223 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486128 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f81d496a-d9ee-4b8c-8275-93cbecf51a8e" containerName="revoke-keys" Jul 17 18:45:45.486223 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486140 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f81d496a-d9ee-4b8c-8275-93cbecf51a8e" containerName="revoke-keys" Jul 17 18:45:45.486223 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486149 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546" containerName="revoke-keys" Jul 17 18:45:45.486223 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486155 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546" containerName="revoke-keys" Jul 17 18:45:45.486223 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486160 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94826487-b1dd-4242-8600-b82d74095fb8" containerName="revoke-keys" Jul 17 18:45:45.486223 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486166 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="94826487-b1dd-4242-8600-b82d74095fb8" containerName="revoke-keys" Jul 17 18:45:45.486223 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486178 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f209f555-739d-47df-9996-90e1afade4c1" containerName="revoke-keys" Jul 17 18:45:45.486223 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486183 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f209f555-739d-47df-9996-90e1afade4c1" containerName="revoke-keys" Jul 17 18:45:45.486223 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486190 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f81d496a-d9ee-4b8c-8275-93cbecf51a8e" containerName="revoke-keys" Jul 17 18:45:45.486223 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486195 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f81d496a-d9ee-4b8c-8275-93cbecf51a8e" containerName="revoke-keys" Jul 17 18:45:45.486223 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486202 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94826487-b1dd-4242-8600-b82d74095fb8" containerName="revoke-keys" Jul 17 18:45:45.486223 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486208 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="94826487-b1dd-4242-8600-b82d74095fb8" containerName="revoke-keys" Jul 17 18:45:45.486223 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486222 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546" containerName="revoke-keys" Jul 17 18:45:45.486223 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486227 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546" containerName="revoke-keys" Jul 17 18:45:45.486641 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486269 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f81d496a-d9ee-4b8c-8275-93cbecf51a8e" containerName="revoke-keys" Jul 17 18:45:45.486641 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486276 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="94826487-b1dd-4242-8600-b82d74095fb8" containerName="revoke-keys" Jul 17 18:45:45.486641 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486283 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="94826487-b1dd-4242-8600-b82d74095fb8" containerName="revoke-keys" Jul 17 18:45:45.486641 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486287 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546" containerName="revoke-keys" Jul 17 18:45:45.486641 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486292 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f81d496a-d9ee-4b8c-8275-93cbecf51a8e" containerName="revoke-keys" Jul 17 18:45:45.486641 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486299 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f209f555-739d-47df-9996-90e1afade4c1" containerName="revoke-keys" Jul 17 18:45:45.486641 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486304 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546" containerName="revoke-keys" Jul 17 18:45:45.486641 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.486308 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546" containerName="revoke-keys" Jul 17 18:45:45.490721 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.490705 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf" Jul 17 18:45:45.493103 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.493078 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:45:45.493245 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.493173 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:45:45.496432 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.496409 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf"] Jul 17 18:45:45.556658 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.556633 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c629c8b3-4f84-49f7-93ea-cab5ab61fb8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf\" (UID: \"c629c8b3-4f84-49f7-93ea-cab5ab61fb8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf" Jul 17 18:45:45.657268 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.657233 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c629c8b3-4f84-49f7-93ea-cab5ab61fb8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf\" (UID: \"c629c8b3-4f84-49f7-93ea-cab5ab61fb8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf" Jul 17 18:45:45.657820 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.657802 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c629c8b3-4f84-49f7-93ea-cab5ab61fb8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf\" (UID: \"c629c8b3-4f84-49f7-93ea-cab5ab61fb8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf" Jul 17 18:45:45.802884 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.802859 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf" Jul 17 18:45:45.931856 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:45.931834 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf"] Jul 17 18:45:45.933627 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:45:45.933596 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc629c8b3_4f84_49f7_93ea_cab5ab61fb8a.slice/crio-5c8b643781e52c361f451246e2f0d38143cc70114f1ec4b04bc80ef1c85fca16 WatchSource:0}: Error finding container 5c8b643781e52c361f451246e2f0d38143cc70114f1ec4b04bc80ef1c85fca16: Status 404 returned error can't find the container with id 5c8b643781e52c361f451246e2f0d38143cc70114f1ec4b04bc80ef1c85fca16 Jul 17 18:45:46.768978 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:46.768941 2572 generic.go:356] "Generic (PLEG): container finished" podID="c629c8b3-4f84-49f7-93ea-cab5ab61fb8a" containerID="dff266f4bd498f84be39685865b928d32dcf34793c39fb85b8112f5cc7f7e6d8" exitCode=6 Jul 17 18:45:46.769354 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:46.769031 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf" event={"ID":"c629c8b3-4f84-49f7-93ea-cab5ab61fb8a","Type":"ContainerDied","Data":"dff266f4bd498f84be39685865b928d32dcf34793c39fb85b8112f5cc7f7e6d8"} Jul 17 18:45:46.769354 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:46.769115 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf" event={"ID":"c629c8b3-4f84-49f7-93ea-cab5ab61fb8a","Type":"ContainerStarted","Data":"5c8b643781e52c361f451246e2f0d38143cc70114f1ec4b04bc80ef1c85fca16"} Jul 17 18:45:46.769354 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:46.769177 2572 scope.go:117] "RemoveContainer" containerID="dff266f4bd498f84be39685865b928d32dcf34793c39fb85b8112f5cc7f7e6d8" Jul 17 18:45:47.774832 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:47.774744 2572 generic.go:356] "Generic (PLEG): container finished" podID="c629c8b3-4f84-49f7-93ea-cab5ab61fb8a" containerID="3d4f0bced83526a3d6d2f6c11311c313aea8f34b9c1b0c34d6efecc57442e890" exitCode=6 Jul 17 18:45:47.775295 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:47.774827 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf" event={"ID":"c629c8b3-4f84-49f7-93ea-cab5ab61fb8a","Type":"ContainerDied","Data":"3d4f0bced83526a3d6d2f6c11311c313aea8f34b9c1b0c34d6efecc57442e890"} Jul 17 18:45:47.775295 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:47.774902 2572 scope.go:117] "RemoveContainer" containerID="dff266f4bd498f84be39685865b928d32dcf34793c39fb85b8112f5cc7f7e6d8" Jul 17 18:45:47.775295 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:47.775105 2572 scope.go:117] "RemoveContainer" containerID="3d4f0bced83526a3d6d2f6c11311c313aea8f34b9c1b0c34d6efecc57442e890" Jul 17 18:45:47.775295 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:45:47.775285 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf_odh-ai-gateway-infra(c629c8b3-4f84-49f7-93ea-cab5ab61fb8a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf" podUID="c629c8b3-4f84-49f7-93ea-cab5ab61fb8a" Jul 17 18:45:48.780679 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:48.780656 2572 scope.go:117] "RemoveContainer" containerID="3d4f0bced83526a3d6d2f6c11311c313aea8f34b9c1b0c34d6efecc57442e890" Jul 17 18:45:48.781156 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:45:48.780838 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf_odh-ai-gateway-infra(c629c8b3-4f84-49f7-93ea-cab5ab61fb8a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf" podUID="c629c8b3-4f84-49f7-93ea-cab5ab61fb8a" Jul 17 18:45:57.615750 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:57.615714 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77"] Jul 17 18:45:57.616153 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:57.616028 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94826487-b1dd-4242-8600-b82d74095fb8" containerName="revoke-keys" Jul 17 18:45:57.616153 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:57.616040 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="94826487-b1dd-4242-8600-b82d74095fb8" containerName="revoke-keys" Jul 17 18:45:57.616153 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:57.616063 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546" containerName="revoke-keys" Jul 17 18:45:57.616153 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:57.616068 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ca5c3d-3d3f-4811-bd2a-b13ea9ce8546" containerName="revoke-keys" Jul 17 18:45:57.616153 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:57.616077 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f81d496a-d9ee-4b8c-8275-93cbecf51a8e" containerName="revoke-keys" Jul 17 18:45:57.616153 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:57.616083 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f81d496a-d9ee-4b8c-8275-93cbecf51a8e" containerName="revoke-keys" Jul 17 18:45:57.616153 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:57.616131 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="94826487-b1dd-4242-8600-b82d74095fb8" containerName="revoke-keys" Jul 17 18:45:57.616153 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:57.616139 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f81d496a-d9ee-4b8c-8275-93cbecf51a8e" containerName="revoke-keys" Jul 17 18:45:57.618860 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:57.618843 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77" Jul 17 18:45:57.626765 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:57.626738 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77"] Jul 17 18:45:57.758390 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:57.758354 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f99f85b-b685-4155-9b79-14ed83039f38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77\" (UID: \"4f99f85b-b685-4155-9b79-14ed83039f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77" Jul 17 18:45:57.859569 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:57.859531 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f99f85b-b685-4155-9b79-14ed83039f38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77\" (UID: \"4f99f85b-b685-4155-9b79-14ed83039f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77" Jul 17 18:45:57.860196 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:57.860174 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f99f85b-b685-4155-9b79-14ed83039f38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77\" (UID: \"4f99f85b-b685-4155-9b79-14ed83039f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77" Jul 17 18:45:57.930484 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:57.930389 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77" Jul 17 18:45:58.053166 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:58.053139 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77"] Jul 17 18:45:58.055173 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:45:58.055139 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f99f85b_b685_4155_9b79_14ed83039f38.slice/crio-bae2386bd4e2f50650c53be4f9121ad33b6725e34a9b16aa2907a7c390b29530 WatchSource:0}: Error finding container bae2386bd4e2f50650c53be4f9121ad33b6725e34a9b16aa2907a7c390b29530: Status 404 returned error can't find the container with id bae2386bd4e2f50650c53be4f9121ad33b6725e34a9b16aa2907a7c390b29530 Jul 17 18:45:58.823918 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:58.823887 2572 generic.go:356] "Generic (PLEG): container finished" podID="4f99f85b-b685-4155-9b79-14ed83039f38" containerID="02766cd8e890f42bb116dd140c61ea4711878aff1f06bc4db1b52d0cb7a6d523" exitCode=6 Jul 17 18:45:58.824296 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:58.823965 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77" event={"ID":"4f99f85b-b685-4155-9b79-14ed83039f38","Type":"ContainerDied","Data":"02766cd8e890f42bb116dd140c61ea4711878aff1f06bc4db1b52d0cb7a6d523"} Jul 17 18:45:58.824296 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:58.824004 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77" event={"ID":"4f99f85b-b685-4155-9b79-14ed83039f38","Type":"ContainerStarted","Data":"bae2386bd4e2f50650c53be4f9121ad33b6725e34a9b16aa2907a7c390b29530"} Jul 17 18:45:58.824296 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:58.824138 2572 scope.go:117] "RemoveContainer" containerID="02766cd8e890f42bb116dd140c61ea4711878aff1f06bc4db1b52d0cb7a6d523" Jul 17 18:45:59.829817 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:59.829742 2572 generic.go:356] "Generic (PLEG): container finished" podID="4f99f85b-b685-4155-9b79-14ed83039f38" containerID="428934ed897a5ea6b88fb1f3b2dc81f74b45da5eea7fee7436b7aed6290ad954" exitCode=6 Jul 17 18:45:59.830195 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:59.829822 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77" event={"ID":"4f99f85b-b685-4155-9b79-14ed83039f38","Type":"ContainerDied","Data":"428934ed897a5ea6b88fb1f3b2dc81f74b45da5eea7fee7436b7aed6290ad954"} Jul 17 18:45:59.830195 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:59.829885 2572 scope.go:117] "RemoveContainer" containerID="02766cd8e890f42bb116dd140c61ea4711878aff1f06bc4db1b52d0cb7a6d523" Jul 17 18:45:59.830195 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:45:59.830111 2572 scope.go:117] "RemoveContainer" containerID="428934ed897a5ea6b88fb1f3b2dc81f74b45da5eea7fee7436b7aed6290ad954" Jul 17 18:45:59.830334 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:45:59.830315 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77_odh-ai-gateway-infra(4f99f85b-b685-4155-9b79-14ed83039f38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77" podUID="4f99f85b-b685-4155-9b79-14ed83039f38" Jul 17 18:46:00.835498 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:00.835469 2572 scope.go:117] "RemoveContainer" containerID="428934ed897a5ea6b88fb1f3b2dc81f74b45da5eea7fee7436b7aed6290ad954" Jul 17 18:46:00.835879 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:46:00.835628 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77_odh-ai-gateway-infra(4f99f85b-b685-4155-9b79-14ed83039f38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77" podUID="4f99f85b-b685-4155-9b79-14ed83039f38" Jul 17 18:46:01.099000 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:01.098910 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf"] Jul 17 18:46:01.103340 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:01.103319 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf" Jul 17 18:46:01.113130 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:01.113098 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf"] Jul 17 18:46:01.188651 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:01.188624 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8397ae-1ade-406a-bf1b-40906bf5d57a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf\" (UID: \"2c8397ae-1ade-406a-bf1b-40906bf5d57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf" Jul 17 18:46:01.289759 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:01.289727 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8397ae-1ade-406a-bf1b-40906bf5d57a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf\" (UID: \"2c8397ae-1ade-406a-bf1b-40906bf5d57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf" Jul 17 18:46:01.290439 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:01.290415 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8397ae-1ade-406a-bf1b-40906bf5d57a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf\" (UID: \"2c8397ae-1ade-406a-bf1b-40906bf5d57a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf" Jul 17 18:46:01.420976 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:01.420900 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf" Jul 17 18:46:01.550323 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:01.550297 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf"] Jul 17 18:46:01.552544 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:46:01.552519 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8397ae_1ade_406a_bf1b_40906bf5d57a.slice/crio-c774599d64602d56145ecdea58848287a021a05374467707d520bf20e60b4b7c WatchSource:0}: Error finding container c774599d64602d56145ecdea58848287a021a05374467707d520bf20e60b4b7c: Status 404 returned error can't find the container with id c774599d64602d56145ecdea58848287a021a05374467707d520bf20e60b4b7c Jul 17 18:46:01.844741 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:01.844709 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf" event={"ID":"2c8397ae-1ade-406a-bf1b-40906bf5d57a","Type":"ContainerStarted","Data":"4c2eb3cf21b417530e740ae559dabbcffc63f33c0c6803fc981eb47dd3d5460f"} Jul 17 18:46:01.845190 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:01.844762 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf" event={"ID":"2c8397ae-1ade-406a-bf1b-40906bf5d57a","Type":"ContainerStarted","Data":"c774599d64602d56145ecdea58848287a021a05374467707d520bf20e60b4b7c"} Jul 17 18:46:01.861366 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:01.861321 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf" podStartSLOduration=0.861303068 podStartE2EDuration="861.303068ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:46:01.859946525 +0000 UTC m=+2957.269374131" watchObservedRunningTime="2026-07-17 18:46:01.861303068 +0000 UTC m=+2957.270730674" Jul 17 18:46:02.220085 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:02.220059 2572 scope.go:117] "RemoveContainer" containerID="3d4f0bced83526a3d6d2f6c11311c313aea8f34b9c1b0c34d6efecc57442e890" Jul 17 18:46:02.851067 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:02.851041 2572 generic.go:356] "Generic (PLEG): container finished" podID="2c8397ae-1ade-406a-bf1b-40906bf5d57a" containerID="4c2eb3cf21b417530e740ae559dabbcffc63f33c0c6803fc981eb47dd3d5460f" exitCode=6 Jul 17 18:46:02.851394 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:02.851116 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf" event={"ID":"2c8397ae-1ade-406a-bf1b-40906bf5d57a","Type":"ContainerDied","Data":"4c2eb3cf21b417530e740ae559dabbcffc63f33c0c6803fc981eb47dd3d5460f"} Jul 17 18:46:02.851394 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:02.851303 2572 scope.go:117] "RemoveContainer" containerID="4c2eb3cf21b417530e740ae559dabbcffc63f33c0c6803fc981eb47dd3d5460f" Jul 17 18:46:02.852908 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:02.852887 2572 generic.go:356] "Generic (PLEG): container finished" podID="c629c8b3-4f84-49f7-93ea-cab5ab61fb8a" containerID="ee1e309943fe099e93c7ef4d77fa4d561ed92648f7c14fdc17d289024c9084c1" exitCode=6 Jul 17 18:46:02.853025 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:02.852945 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf" event={"ID":"c629c8b3-4f84-49f7-93ea-cab5ab61fb8a","Type":"ContainerDied","Data":"ee1e309943fe099e93c7ef4d77fa4d561ed92648f7c14fdc17d289024c9084c1"} Jul 17 18:46:02.853025 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:02.852981 2572 scope.go:117] "RemoveContainer" containerID="3d4f0bced83526a3d6d2f6c11311c313aea8f34b9c1b0c34d6efecc57442e890" Jul 17 18:46:02.853229 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:02.853210 2572 scope.go:117] "RemoveContainer" containerID="ee1e309943fe099e93c7ef4d77fa4d561ed92648f7c14fdc17d289024c9084c1" Jul 17 18:46:02.853461 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:46:02.853411 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf_odh-ai-gateway-infra(c629c8b3-4f84-49f7-93ea-cab5ab61fb8a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf" podUID="c629c8b3-4f84-49f7-93ea-cab5ab61fb8a" Jul 17 18:46:03.642603 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:03.642570 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj"] Jul 17 18:46:03.645767 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:03.645748 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj" Jul 17 18:46:03.653640 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:03.653616 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj"] Jul 17 18:46:03.816339 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:03.816303 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c644f8-a510-494c-b1be-66a82f355736-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-x89rj\" (UID: \"a7c644f8-a510-494c-b1be-66a82f355736\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj" Jul 17 18:46:03.859286 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:03.859253 2572 generic.go:356] "Generic (PLEG): container finished" podID="2c8397ae-1ade-406a-bf1b-40906bf5d57a" containerID="22b56c01986dab6daa1c6255792a04849fc1c6eaaf4045d8b6f06f8a02f11530" exitCode=6 Jul 17 18:46:03.859719 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:03.859351 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf" event={"ID":"2c8397ae-1ade-406a-bf1b-40906bf5d57a","Type":"ContainerDied","Data":"22b56c01986dab6daa1c6255792a04849fc1c6eaaf4045d8b6f06f8a02f11530"} Jul 17 18:46:03.859719 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:03.859429 2572 scope.go:117] "RemoveContainer" containerID="4c2eb3cf21b417530e740ae559dabbcffc63f33c0c6803fc981eb47dd3d5460f" Jul 17 18:46:03.859719 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:03.859626 2572 scope.go:117] "RemoveContainer" containerID="22b56c01986dab6daa1c6255792a04849fc1c6eaaf4045d8b6f06f8a02f11530" Jul 17 18:46:03.859882 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:46:03.859863 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf_odh-ai-gateway-infra(2c8397ae-1ade-406a-bf1b-40906bf5d57a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf" podUID="2c8397ae-1ade-406a-bf1b-40906bf5d57a" Jul 17 18:46:03.890782 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:03.890755 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf"] Jul 17 18:46:03.917078 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:03.917046 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c644f8-a510-494c-b1be-66a82f355736-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-x89rj\" (UID: \"a7c644f8-a510-494c-b1be-66a82f355736\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj" Jul 17 18:46:03.917909 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:03.917879 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c644f8-a510-494c-b1be-66a82f355736-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-x89rj\" (UID: \"a7c644f8-a510-494c-b1be-66a82f355736\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj" Jul 17 18:46:03.958429 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:03.958397 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj" Jul 17 18:46:04.038622 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:04.038599 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf" Jul 17 18:46:04.105088 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:04.105062 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj"] Jul 17 18:46:04.107492 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:46:04.107463 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c644f8_a510_494c_b1be_66a82f355736.slice/crio-fb671bd6610e3d153ba2f1e8d26acbf175ed8b24efea1d2924f1a855b15ddf03 WatchSource:0}: Error finding container fb671bd6610e3d153ba2f1e8d26acbf175ed8b24efea1d2924f1a855b15ddf03: Status 404 returned error can't find the container with id fb671bd6610e3d153ba2f1e8d26acbf175ed8b24efea1d2924f1a855b15ddf03 Jul 17 18:46:04.119178 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:04.119157 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c629c8b3-4f84-49f7-93ea-cab5ab61fb8a-maas-api-service-ca\") pod \"c629c8b3-4f84-49f7-93ea-cab5ab61fb8a\" (UID: \"c629c8b3-4f84-49f7-93ea-cab5ab61fb8a\") " Jul 17 18:46:04.119500 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:04.119480 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c629c8b3-4f84-49f7-93ea-cab5ab61fb8a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c629c8b3-4f84-49f7-93ea-cab5ab61fb8a" (UID: "c629c8b3-4f84-49f7-93ea-cab5ab61fb8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:46:04.220114 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:04.220083 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c629c8b3-4f84-49f7-93ea-cab5ab61fb8a-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:46:04.872563 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:04.872538 2572 scope.go:117] "RemoveContainer" containerID="22b56c01986dab6daa1c6255792a04849fc1c6eaaf4045d8b6f06f8a02f11530" Jul 17 18:46:04.872975 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:46:04.872783 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf_odh-ai-gateway-infra(2c8397ae-1ade-406a-bf1b-40906bf5d57a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf" podUID="2c8397ae-1ade-406a-bf1b-40906bf5d57a" Jul 17 18:46:04.873724 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:04.873700 2572 generic.go:356] "Generic (PLEG): container finished" podID="a7c644f8-a510-494c-b1be-66a82f355736" containerID="ba08786a8c971330a4ac7c753ba12f06742ee93e542a206a3b8f22a25b4adf14" exitCode=6 Jul 17 18:46:04.873851 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:04.873779 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj" event={"ID":"a7c644f8-a510-494c-b1be-66a82f355736","Type":"ContainerDied","Data":"ba08786a8c971330a4ac7c753ba12f06742ee93e542a206a3b8f22a25b4adf14"} Jul 17 18:46:04.873916 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:04.873851 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj" event={"ID":"a7c644f8-a510-494c-b1be-66a82f355736","Type":"ContainerStarted","Data":"fb671bd6610e3d153ba2f1e8d26acbf175ed8b24efea1d2924f1a855b15ddf03"} Jul 17 18:46:04.873916 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:04.873909 2572 scope.go:117] "RemoveContainer" containerID="ba08786a8c971330a4ac7c753ba12f06742ee93e542a206a3b8f22a25b4adf14" Jul 17 18:46:04.875115 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:04.875091 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf" event={"ID":"c629c8b3-4f84-49f7-93ea-cab5ab61fb8a","Type":"ContainerDied","Data":"5c8b643781e52c361f451246e2f0d38143cc70114f1ec4b04bc80ef1c85fca16"} Jul 17 18:46:04.875115 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:04.875111 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf" Jul 17 18:46:04.875235 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:04.875134 2572 scope.go:117] "RemoveContainer" containerID="ee1e309943fe099e93c7ef4d77fa4d561ed92648f7c14fdc17d289024c9084c1" Jul 17 18:46:04.913838 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:04.913807 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf"] Jul 17 18:46:04.919110 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:04.919080 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-wtvrf"] Jul 17 18:46:05.225378 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:05.225300 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c629c8b3-4f84-49f7-93ea-cab5ab61fb8a" path="/var/lib/kubelet/pods/c629c8b3-4f84-49f7-93ea-cab5ab61fb8a/volumes" Jul 17 18:46:05.881802 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:05.881714 2572 generic.go:356] "Generic (PLEG): container finished" podID="a7c644f8-a510-494c-b1be-66a82f355736" containerID="5c301177426a991b2eb4aba2abf17b6bcb49ed33cd14f11b38fe799a88d4f25a" exitCode=6 Jul 17 18:46:05.881802 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:05.881792 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj" event={"ID":"a7c644f8-a510-494c-b1be-66a82f355736","Type":"ContainerDied","Data":"5c301177426a991b2eb4aba2abf17b6bcb49ed33cd14f11b38fe799a88d4f25a"} Jul 17 18:46:05.882337 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:05.881849 2572 scope.go:117] "RemoveContainer" containerID="ba08786a8c971330a4ac7c753ba12f06742ee93e542a206a3b8f22a25b4adf14" Jul 17 18:46:05.882337 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:05.882044 2572 scope.go:117] "RemoveContainer" containerID="5c301177426a991b2eb4aba2abf17b6bcb49ed33cd14f11b38fe799a88d4f25a" Jul 17 18:46:05.882337 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:46:05.882295 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-x89rj_odh-ai-gateway-infra(a7c644f8-a510-494c-b1be-66a82f355736)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj" podUID="a7c644f8-a510-494c-b1be-66a82f355736" Jul 17 18:46:06.888858 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:06.888832 2572 scope.go:117] "RemoveContainer" containerID="5c301177426a991b2eb4aba2abf17b6bcb49ed33cd14f11b38fe799a88d4f25a" Jul 17 18:46:06.889264 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:46:06.889002 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-x89rj_odh-ai-gateway-infra(a7c644f8-a510-494c-b1be-66a82f355736)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj" podUID="a7c644f8-a510-494c-b1be-66a82f355736" Jul 17 18:46:13.219827 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:13.219793 2572 scope.go:117] "RemoveContainer" containerID="428934ed897a5ea6b88fb1f3b2dc81f74b45da5eea7fee7436b7aed6290ad954" Jul 17 18:46:13.926124 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:13.926040 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77" event={"ID":"4f99f85b-b685-4155-9b79-14ed83039f38","Type":"ContainerStarted","Data":"2c8f66aaadb3eac239f829623aef3bccad5fd738bf88dc7538b4a0c12c6dd233"} Jul 17 18:46:13.944379 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:13.944337 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77" podStartSLOduration=16.944321044 podStartE2EDuration="16.944321044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:46:13.942473821 +0000 UTC m=+2969.351901449" watchObservedRunningTime="2026-07-17 18:46:13.944321044 +0000 UTC m=+2969.353748700" Jul 17 18:46:14.932842 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:14.932811 2572 generic.go:356] "Generic (PLEG): container finished" podID="4f99f85b-b685-4155-9b79-14ed83039f38" containerID="2c8f66aaadb3eac239f829623aef3bccad5fd738bf88dc7538b4a0c12c6dd233" exitCode=6 Jul 17 18:46:14.933303 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:14.932892 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77" event={"ID":"4f99f85b-b685-4155-9b79-14ed83039f38","Type":"ContainerDied","Data":"2c8f66aaadb3eac239f829623aef3bccad5fd738bf88dc7538b4a0c12c6dd233"} Jul 17 18:46:14.933303 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:14.932955 2572 scope.go:117] "RemoveContainer" containerID="428934ed897a5ea6b88fb1f3b2dc81f74b45da5eea7fee7436b7aed6290ad954" Jul 17 18:46:14.933303 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:14.933264 2572 scope.go:117] "RemoveContainer" containerID="2c8f66aaadb3eac239f829623aef3bccad5fd738bf88dc7538b4a0c12c6dd233" Jul 17 18:46:14.933484 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:46:14.933445 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77_odh-ai-gateway-infra(4f99f85b-b685-4155-9b79-14ed83039f38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77" podUID="4f99f85b-b685-4155-9b79-14ed83039f38" Jul 17 18:46:14.960040 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:14.960002 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77"] Jul 17 18:46:15.223221 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:15.223147 2572 scope.go:117] "RemoveContainer" containerID="22b56c01986dab6daa1c6255792a04849fc1c6eaaf4045d8b6f06f8a02f11530" Jul 17 18:46:15.938230 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:15.938193 2572 generic.go:356] "Generic (PLEG): container finished" podID="2c8397ae-1ade-406a-bf1b-40906bf5d57a" containerID="1f3d872cb854036eb532309004bb20466ef0fa36a46f54819118435d53035dba" exitCode=6 Jul 17 18:46:15.938648 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:15.938269 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf" event={"ID":"2c8397ae-1ade-406a-bf1b-40906bf5d57a","Type":"ContainerDied","Data":"1f3d872cb854036eb532309004bb20466ef0fa36a46f54819118435d53035dba"} Jul 17 18:46:15.938648 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:15.938351 2572 scope.go:117] "RemoveContainer" containerID="22b56c01986dab6daa1c6255792a04849fc1c6eaaf4045d8b6f06f8a02f11530" Jul 17 18:46:15.938760 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:15.938647 2572 scope.go:117] "RemoveContainer" containerID="1f3d872cb854036eb532309004bb20466ef0fa36a46f54819118435d53035dba" Jul 17 18:46:15.938900 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:46:15.938871 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf_odh-ai-gateway-infra(2c8397ae-1ade-406a-bf1b-40906bf5d57a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf" podUID="2c8397ae-1ade-406a-bf1b-40906bf5d57a" Jul 17 18:46:16.068482 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:16.068458 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77" Jul 17 18:46:16.226071 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:16.226034 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f99f85b-b685-4155-9b79-14ed83039f38-maas-api-service-ca\") pod \"4f99f85b-b685-4155-9b79-14ed83039f38\" (UID: \"4f99f85b-b685-4155-9b79-14ed83039f38\") " Jul 17 18:46:16.226352 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:16.226330 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f99f85b-b685-4155-9b79-14ed83039f38-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4f99f85b-b685-4155-9b79-14ed83039f38" (UID: "4f99f85b-b685-4155-9b79-14ed83039f38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:46:16.327448 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:16.327417 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f99f85b-b685-4155-9b79-14ed83039f38-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:46:16.945821 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:16.945786 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77" Jul 17 18:46:16.946303 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:16.945820 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77" event={"ID":"4f99f85b-b685-4155-9b79-14ed83039f38","Type":"ContainerDied","Data":"bae2386bd4e2f50650c53be4f9121ad33b6725e34a9b16aa2907a7c390b29530"} Jul 17 18:46:16.946303 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:16.945886 2572 scope.go:117] "RemoveContainer" containerID="2c8f66aaadb3eac239f829623aef3bccad5fd738bf88dc7538b4a0c12c6dd233" Jul 17 18:46:16.964552 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:16.964524 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf"] Jul 17 18:46:16.968576 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:16.968553 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77"] Jul 17 18:46:16.975985 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:16.975966 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-n9h77"] Jul 17 18:46:17.091215 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:17.091194 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf" Jul 17 18:46:17.133172 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:17.133149 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8397ae-1ade-406a-bf1b-40906bf5d57a-maas-api-service-ca\") pod \"2c8397ae-1ade-406a-bf1b-40906bf5d57a\" (UID: \"2c8397ae-1ade-406a-bf1b-40906bf5d57a\") " Jul 17 18:46:17.133684 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:17.133654 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8397ae-1ade-406a-bf1b-40906bf5d57a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c8397ae-1ade-406a-bf1b-40906bf5d57a" (UID: "2c8397ae-1ade-406a-bf1b-40906bf5d57a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:46:17.224599 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:17.224523 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f99f85b-b685-4155-9b79-14ed83039f38" path="/var/lib/kubelet/pods/4f99f85b-b685-4155-9b79-14ed83039f38/volumes" Jul 17 18:46:17.233617 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:17.233598 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8397ae-1ade-406a-bf1b-40906bf5d57a-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:46:17.953550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:17.953517 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf" event={"ID":"2c8397ae-1ade-406a-bf1b-40906bf5d57a","Type":"ContainerDied","Data":"c774599d64602d56145ecdea58848287a021a05374467707d520bf20e60b4b7c"} Jul 17 18:46:17.953550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:17.953540 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf" Jul 17 18:46:17.954066 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:17.953580 2572 scope.go:117] "RemoveContainer" containerID="1f3d872cb854036eb532309004bb20466ef0fa36a46f54819118435d53035dba" Jul 17 18:46:17.968538 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:17.968515 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf"] Jul 17 18:46:17.975081 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:17.975056 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-zsxwf"] Jul 17 18:46:19.224438 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:19.224408 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c8397ae-1ade-406a-bf1b-40906bf5d57a" path="/var/lib/kubelet/pods/2c8397ae-1ade-406a-bf1b-40906bf5d57a/volumes" Jul 17 18:46:20.219482 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:20.219448 2572 scope.go:117] "RemoveContainer" containerID="5c301177426a991b2eb4aba2abf17b6bcb49ed33cd14f11b38fe799a88d4f25a" Jul 17 18:46:20.970100 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:20.970066 2572 generic.go:356] "Generic (PLEG): container finished" podID="a7c644f8-a510-494c-b1be-66a82f355736" containerID="383b504844eb9ae42af72bdb927b94e814176d27a18e711953746948d693ee89" exitCode=6 Jul 17 18:46:20.970466 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:20.970141 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj" event={"ID":"a7c644f8-a510-494c-b1be-66a82f355736","Type":"ContainerDied","Data":"383b504844eb9ae42af72bdb927b94e814176d27a18e711953746948d693ee89"} Jul 17 18:46:20.970466 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:20.970228 2572 scope.go:117] "RemoveContainer" containerID="5c301177426a991b2eb4aba2abf17b6bcb49ed33cd14f11b38fe799a88d4f25a" Jul 17 18:46:20.970583 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:20.970568 2572 scope.go:117] "RemoveContainer" containerID="383b504844eb9ae42af72bdb927b94e814176d27a18e711953746948d693ee89" Jul 17 18:46:20.970773 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:46:20.970755 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-x89rj_odh-ai-gateway-infra(a7c644f8-a510-494c-b1be-66a82f355736)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj" podUID="a7c644f8-a510-494c-b1be-66a82f355736" Jul 17 18:46:21.995050 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:21.994999 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj"] Jul 17 18:46:22.122154 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:22.122131 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj" Jul 17 18:46:22.171507 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:22.171474 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c644f8-a510-494c-b1be-66a82f355736-maas-api-service-ca\") pod \"a7c644f8-a510-494c-b1be-66a82f355736\" (UID: \"a7c644f8-a510-494c-b1be-66a82f355736\") " Jul 17 18:46:22.171828 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:22.171804 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c644f8-a510-494c-b1be-66a82f355736-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a7c644f8-a510-494c-b1be-66a82f355736" (UID: "a7c644f8-a510-494c-b1be-66a82f355736"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:46:22.272041 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:22.271941 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7c644f8-a510-494c-b1be-66a82f355736-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:46:22.981646 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:22.981609 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj" event={"ID":"a7c644f8-a510-494c-b1be-66a82f355736","Type":"ContainerDied","Data":"fb671bd6610e3d153ba2f1e8d26acbf175ed8b24efea1d2924f1a855b15ddf03"} Jul 17 18:46:22.981646 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:22.981641 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj" Jul 17 18:46:22.981912 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:22.981677 2572 scope.go:117] "RemoveContainer" containerID="383b504844eb9ae42af72bdb927b94e814176d27a18e711953746948d693ee89" Jul 17 18:46:23.002402 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:23.002375 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj"] Jul 17 18:46:23.007699 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:23.007671 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-x89rj"] Jul 17 18:46:23.225177 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:23.225146 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7c644f8-a510-494c-b1be-66a82f355736" path="/var/lib/kubelet/pods/a7c644f8-a510-494c-b1be-66a82f355736/volumes" Jul 17 18:46:35.532614 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.532572 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94"] Jul 17 18:46:35.533237 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.532983 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c8397ae-1ade-406a-bf1b-40906bf5d57a" containerName="revoke-keys" Jul 17 18:46:35.533237 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533000 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c8397ae-1ade-406a-bf1b-40906bf5d57a" containerName="revoke-keys" Jul 17 18:46:35.533237 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533028 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c8397ae-1ade-406a-bf1b-40906bf5d57a" containerName="revoke-keys" Jul 17 18:46:35.533237 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533038 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c8397ae-1ade-406a-bf1b-40906bf5d57a" containerName="revoke-keys" Jul 17 18:46:35.533237 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533050 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7c644f8-a510-494c-b1be-66a82f355736" containerName="revoke-keys" Jul 17 18:46:35.533237 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533059 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7c644f8-a510-494c-b1be-66a82f355736" containerName="revoke-keys" Jul 17 18:46:35.533237 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533069 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f99f85b-b685-4155-9b79-14ed83039f38" containerName="revoke-keys" Jul 17 18:46:35.533237 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533076 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f99f85b-b685-4155-9b79-14ed83039f38" containerName="revoke-keys" Jul 17 18:46:35.533237 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533086 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7c644f8-a510-494c-b1be-66a82f355736" containerName="revoke-keys" Jul 17 18:46:35.533237 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533094 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7c644f8-a510-494c-b1be-66a82f355736" containerName="revoke-keys" Jul 17 18:46:35.533237 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533114 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c629c8b3-4f84-49f7-93ea-cab5ab61fb8a" containerName="revoke-keys" Jul 17 18:46:35.533237 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533123 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c629c8b3-4f84-49f7-93ea-cab5ab61fb8a" containerName="revoke-keys" Jul 17 18:46:35.533237 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533138 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c629c8b3-4f84-49f7-93ea-cab5ab61fb8a" containerName="revoke-keys" Jul 17 18:46:35.533237 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533146 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c629c8b3-4f84-49f7-93ea-cab5ab61fb8a" containerName="revoke-keys" Jul 17 18:46:35.533237 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533159 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c629c8b3-4f84-49f7-93ea-cab5ab61fb8a" containerName="revoke-keys" Jul 17 18:46:35.533237 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533167 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c629c8b3-4f84-49f7-93ea-cab5ab61fb8a" containerName="revoke-keys" Jul 17 18:46:35.533237 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533186 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f99f85b-b685-4155-9b79-14ed83039f38" containerName="revoke-keys" Jul 17 18:46:35.533237 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533194 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f99f85b-b685-4155-9b79-14ed83039f38" containerName="revoke-keys" Jul 17 18:46:35.534154 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533271 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c629c8b3-4f84-49f7-93ea-cab5ab61fb8a" containerName="revoke-keys" Jul 17 18:46:35.534154 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533284 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c8397ae-1ade-406a-bf1b-40906bf5d57a" containerName="revoke-keys" Jul 17 18:46:35.534154 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533293 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f99f85b-b685-4155-9b79-14ed83039f38" containerName="revoke-keys" Jul 17 18:46:35.534154 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533302 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f99f85b-b685-4155-9b79-14ed83039f38" containerName="revoke-keys" Jul 17 18:46:35.534154 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533312 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c629c8b3-4f84-49f7-93ea-cab5ab61fb8a" containerName="revoke-keys" Jul 17 18:46:35.534154 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533319 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7c644f8-a510-494c-b1be-66a82f355736" containerName="revoke-keys" Jul 17 18:46:35.534154 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533327 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f99f85b-b685-4155-9b79-14ed83039f38" containerName="revoke-keys" Jul 17 18:46:35.534154 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533335 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7c644f8-a510-494c-b1be-66a82f355736" containerName="revoke-keys" Jul 17 18:46:35.534154 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533344 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c8397ae-1ade-406a-bf1b-40906bf5d57a" containerName="revoke-keys" Jul 17 18:46:35.534154 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.533351 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7c644f8-a510-494c-b1be-66a82f355736" containerName="revoke-keys" Jul 17 18:46:35.536594 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.536572 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94" Jul 17 18:46:35.539204 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.539183 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:46:35.539309 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.539219 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:46:35.544666 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.544642 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94"] Jul 17 18:46:35.586914 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.586872 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ba51fc-e1fe-46ca-b42b-9c23d239782a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94\" (UID: \"09ba51fc-e1fe-46ca-b42b-9c23d239782a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94" Jul 17 18:46:35.687759 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.687719 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ba51fc-e1fe-46ca-b42b-9c23d239782a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94\" (UID: \"09ba51fc-e1fe-46ca-b42b-9c23d239782a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94" Jul 17 18:46:35.688410 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.688385 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ba51fc-e1fe-46ca-b42b-9c23d239782a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94\" (UID: \"09ba51fc-e1fe-46ca-b42b-9c23d239782a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94" Jul 17 18:46:35.848207 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.848119 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94" Jul 17 18:46:35.973571 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:35.973539 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94"] Jul 17 18:46:35.976757 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:46:35.976731 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ba51fc_e1fe_46ca_b42b_9c23d239782a.slice/crio-e764400d72d5ced8f8a9f2ed67ec0d293b70ecf4d27dc0e83d3da7fcdc5e87bf WatchSource:0}: Error finding container e764400d72d5ced8f8a9f2ed67ec0d293b70ecf4d27dc0e83d3da7fcdc5e87bf: Status 404 returned error can't find the container with id e764400d72d5ced8f8a9f2ed67ec0d293b70ecf4d27dc0e83d3da7fcdc5e87bf Jul 17 18:46:36.040966 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:36.040928 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94" event={"ID":"09ba51fc-e1fe-46ca-b42b-9c23d239782a","Type":"ContainerStarted","Data":"e764400d72d5ced8f8a9f2ed67ec0d293b70ecf4d27dc0e83d3da7fcdc5e87bf"} Jul 17 18:46:37.046258 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:37.046227 2572 generic.go:356] "Generic (PLEG): container finished" podID="09ba51fc-e1fe-46ca-b42b-9c23d239782a" containerID="f1562500b953a6bc912829f5e12eb5f3bf54450f3139d1f935e11f1c7355c0cd" exitCode=6 Jul 17 18:46:37.046710 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:37.046320 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94" event={"ID":"09ba51fc-e1fe-46ca-b42b-9c23d239782a","Type":"ContainerDied","Data":"f1562500b953a6bc912829f5e12eb5f3bf54450f3139d1f935e11f1c7355c0cd"} Jul 17 18:46:37.046710 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:37.046472 2572 scope.go:117] "RemoveContainer" containerID="f1562500b953a6bc912829f5e12eb5f3bf54450f3139d1f935e11f1c7355c0cd" Jul 17 18:46:38.052418 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:38.052384 2572 generic.go:356] "Generic (PLEG): container finished" podID="09ba51fc-e1fe-46ca-b42b-9c23d239782a" containerID="6baf3ad2b663263a074eb1ce8ae24f3d4ce131bc44f3c49a87e0c222be08ac60" exitCode=6 Jul 17 18:46:38.052845 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:38.052450 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94" event={"ID":"09ba51fc-e1fe-46ca-b42b-9c23d239782a","Type":"ContainerDied","Data":"6baf3ad2b663263a074eb1ce8ae24f3d4ce131bc44f3c49a87e0c222be08ac60"} Jul 17 18:46:38.052845 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:38.052505 2572 scope.go:117] "RemoveContainer" containerID="f1562500b953a6bc912829f5e12eb5f3bf54450f3139d1f935e11f1c7355c0cd" Jul 17 18:46:38.052845 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:38.052774 2572 scope.go:117] "RemoveContainer" containerID="6baf3ad2b663263a074eb1ce8ae24f3d4ce131bc44f3c49a87e0c222be08ac60" Jul 17 18:46:38.053028 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:46:38.052986 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94_odh-ai-gateway-infra(09ba51fc-e1fe-46ca-b42b-9c23d239782a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94" podUID="09ba51fc-e1fe-46ca-b42b-9c23d239782a" Jul 17 18:46:39.058582 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:39.058557 2572 scope.go:117] "RemoveContainer" containerID="6baf3ad2b663263a074eb1ce8ae24f3d4ce131bc44f3c49a87e0c222be08ac60" Jul 17 18:46:39.058970 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:46:39.058754 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94_odh-ai-gateway-infra(09ba51fc-e1fe-46ca-b42b-9c23d239782a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94" podUID="09ba51fc-e1fe-46ca-b42b-9c23d239782a" Jul 17 18:46:47.658110 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:47.658077 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr"] Jul 17 18:46:47.658501 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:47.658379 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c8397ae-1ade-406a-bf1b-40906bf5d57a" containerName="revoke-keys" Jul 17 18:46:47.658501 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:47.658390 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c8397ae-1ade-406a-bf1b-40906bf5d57a" containerName="revoke-keys" Jul 17 18:46:47.658501 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:47.658397 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f99f85b-b685-4155-9b79-14ed83039f38" containerName="revoke-keys" Jul 17 18:46:47.658501 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:47.658403 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f99f85b-b685-4155-9b79-14ed83039f38" containerName="revoke-keys" Jul 17 18:46:47.658501 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:47.658410 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7c644f8-a510-494c-b1be-66a82f355736" containerName="revoke-keys" Jul 17 18:46:47.658501 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:47.658416 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7c644f8-a510-494c-b1be-66a82f355736" containerName="revoke-keys" Jul 17 18:46:47.658501 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:47.658458 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c8397ae-1ade-406a-bf1b-40906bf5d57a" containerName="revoke-keys" Jul 17 18:46:47.658501 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:47.658468 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c629c8b3-4f84-49f7-93ea-cab5ab61fb8a" containerName="revoke-keys" Jul 17 18:46:47.663178 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:47.663160 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr" Jul 17 18:46:47.667834 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:47.667810 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr"] Jul 17 18:46:47.792626 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:47.792592 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9df574e-e59c-42ef-9966-5257f9697b58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr\" (UID: \"d9df574e-e59c-42ef-9966-5257f9697b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr" Jul 17 18:46:47.893724 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:47.893690 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9df574e-e59c-42ef-9966-5257f9697b58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr\" (UID: \"d9df574e-e59c-42ef-9966-5257f9697b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr" Jul 17 18:46:47.894308 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:47.894291 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9df574e-e59c-42ef-9966-5257f9697b58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr\" (UID: \"d9df574e-e59c-42ef-9966-5257f9697b58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr" Jul 17 18:46:47.976171 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:47.976088 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr" Jul 17 18:46:48.107156 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:48.107128 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr"] Jul 17 18:46:48.108866 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:46:48.108837 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9df574e_e59c_42ef_9966_5257f9697b58.slice/crio-5b1aa8fed766e9303fa503905c63ae150d3274363a115f14aee3d6ec4d500569 WatchSource:0}: Error finding container 5b1aa8fed766e9303fa503905c63ae150d3274363a115f14aee3d6ec4d500569: Status 404 returned error can't find the container with id 5b1aa8fed766e9303fa503905c63ae150d3274363a115f14aee3d6ec4d500569 Jul 17 18:46:49.107526 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:49.107490 2572 generic.go:356] "Generic (PLEG): container finished" podID="d9df574e-e59c-42ef-9966-5257f9697b58" containerID="acfc22935e1ac0fff9187ad08aa99b29c46246ba9a065cc6c62d8cdc70100ca1" exitCode=6 Jul 17 18:46:49.107975 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:49.107570 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr" event={"ID":"d9df574e-e59c-42ef-9966-5257f9697b58","Type":"ContainerDied","Data":"acfc22935e1ac0fff9187ad08aa99b29c46246ba9a065cc6c62d8cdc70100ca1"} Jul 17 18:46:49.107975 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:49.107621 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr" event={"ID":"d9df574e-e59c-42ef-9966-5257f9697b58","Type":"ContainerStarted","Data":"5b1aa8fed766e9303fa503905c63ae150d3274363a115f14aee3d6ec4d500569"} Jul 17 18:46:49.107975 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:49.107722 2572 scope.go:117] "RemoveContainer" containerID="acfc22935e1ac0fff9187ad08aa99b29c46246ba9a065cc6c62d8cdc70100ca1" Jul 17 18:46:50.114140 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:50.114106 2572 generic.go:356] "Generic (PLEG): container finished" podID="d9df574e-e59c-42ef-9966-5257f9697b58" containerID="481df4c8a90af1e0c00c18d35e9657dc7be69876ba1bf054c8519de93e48aed5" exitCode=6 Jul 17 18:46:50.114528 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:50.114192 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr" event={"ID":"d9df574e-e59c-42ef-9966-5257f9697b58","Type":"ContainerDied","Data":"481df4c8a90af1e0c00c18d35e9657dc7be69876ba1bf054c8519de93e48aed5"} Jul 17 18:46:50.114528 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:50.114254 2572 scope.go:117] "RemoveContainer" containerID="acfc22935e1ac0fff9187ad08aa99b29c46246ba9a065cc6c62d8cdc70100ca1" Jul 17 18:46:50.114528 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:50.114517 2572 scope.go:117] "RemoveContainer" containerID="481df4c8a90af1e0c00c18d35e9657dc7be69876ba1bf054c8519de93e48aed5" Jul 17 18:46:50.114727 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:46:50.114700 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr_odh-ai-gateway-infra(d9df574e-e59c-42ef-9966-5257f9697b58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr" podUID="d9df574e-e59c-42ef-9966-5257f9697b58" Jul 17 18:46:51.124867 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:51.124839 2572 scope.go:117] "RemoveContainer" containerID="481df4c8a90af1e0c00c18d35e9657dc7be69876ba1bf054c8519de93e48aed5" Jul 17 18:46:51.125288 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:46:51.125026 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr_odh-ai-gateway-infra(d9df574e-e59c-42ef-9966-5257f9697b58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr" podUID="d9df574e-e59c-42ef-9966-5257f9697b58" Jul 17 18:46:51.142561 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:51.142532 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv"] Jul 17 18:46:51.146921 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:51.146906 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv" Jul 17 18:46:51.153432 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:51.153406 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv"] Jul 17 18:46:51.224857 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:51.224819 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4319849b-e4ac-4c5a-afbe-424ff5d72bfd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv\" (UID: \"4319849b-e4ac-4c5a-afbe-424ff5d72bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv" Jul 17 18:46:51.325528 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:51.325497 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4319849b-e4ac-4c5a-afbe-424ff5d72bfd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv\" (UID: \"4319849b-e4ac-4c5a-afbe-424ff5d72bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv" Jul 17 18:46:51.326106 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:51.326087 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4319849b-e4ac-4c5a-afbe-424ff5d72bfd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv\" (UID: \"4319849b-e4ac-4c5a-afbe-424ff5d72bfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv" Jul 17 18:46:51.459488 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:51.459407 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv" Jul 17 18:46:51.585940 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:51.585917 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv"] Jul 17 18:46:51.587851 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:46:51.587822 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4319849b_e4ac_4c5a_afbe_424ff5d72bfd.slice/crio-510a658df56a5818906e493a1094f2a6da1fbd3b336a0ffe8cfc33a682348387 WatchSource:0}: Error finding container 510a658df56a5818906e493a1094f2a6da1fbd3b336a0ffe8cfc33a682348387: Status 404 returned error can't find the container with id 510a658df56a5818906e493a1094f2a6da1fbd3b336a0ffe8cfc33a682348387 Jul 17 18:46:52.130943 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:52.130910 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv" event={"ID":"4319849b-e4ac-4c5a-afbe-424ff5d72bfd","Type":"ContainerStarted","Data":"9200a9751953ed646b72df9bc6c640ea1bdc9c30564e03fafc9a8c1b44a48b5d"} Jul 17 18:46:52.131350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:52.130961 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv" event={"ID":"4319849b-e4ac-4c5a-afbe-424ff5d72bfd","Type":"ContainerStarted","Data":"510a658df56a5818906e493a1094f2a6da1fbd3b336a0ffe8cfc33a682348387"} Jul 17 18:46:52.146964 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:52.146916 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv" podStartSLOduration=1.146900828 podStartE2EDuration="1.146900828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:46:52.145765 +0000 UTC m=+3007.555192623" watchObservedRunningTime="2026-07-17 18:46:52.146900828 +0000 UTC m=+3007.556328433" Jul 17 18:46:53.137572 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:53.137536 2572 generic.go:356] "Generic (PLEG): container finished" podID="4319849b-e4ac-4c5a-afbe-424ff5d72bfd" containerID="9200a9751953ed646b72df9bc6c640ea1bdc9c30564e03fafc9a8c1b44a48b5d" exitCode=6 Jul 17 18:46:53.138073 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:53.137594 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv" event={"ID":"4319849b-e4ac-4c5a-afbe-424ff5d72bfd","Type":"ContainerDied","Data":"9200a9751953ed646b72df9bc6c640ea1bdc9c30564e03fafc9a8c1b44a48b5d"} Jul 17 18:46:53.138073 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:53.137817 2572 scope.go:117] "RemoveContainer" containerID="9200a9751953ed646b72df9bc6c640ea1bdc9c30564e03fafc9a8c1b44a48b5d" Jul 17 18:46:53.220311 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:53.220285 2572 scope.go:117] "RemoveContainer" containerID="6baf3ad2b663263a074eb1ce8ae24f3d4ce131bc44f3c49a87e0c222be08ac60" Jul 17 18:46:53.683029 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:53.682979 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s"] Jul 17 18:46:53.686205 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:53.686190 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s" Jul 17 18:46:53.693616 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:53.693593 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s"] Jul 17 18:46:53.845976 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:53.845951 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d0c49a3-d545-4377-ac32-7b82a2df0ff1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-k896s\" (UID: \"0d0c49a3-d545-4377-ac32-7b82a2df0ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s" Jul 17 18:46:53.946660 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:53.946580 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d0c49a3-d545-4377-ac32-7b82a2df0ff1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-k896s\" (UID: \"0d0c49a3-d545-4377-ac32-7b82a2df0ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s" Jul 17 18:46:53.947258 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:53.947240 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d0c49a3-d545-4377-ac32-7b82a2df0ff1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-k896s\" (UID: \"0d0c49a3-d545-4377-ac32-7b82a2df0ff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s" Jul 17 18:46:53.997595 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:53.997571 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s" Jul 17 18:46:54.120555 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:54.120456 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s"] Jul 17 18:46:54.122492 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:46:54.122468 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0c49a3_d545_4377_ac32_7b82a2df0ff1.slice/crio-e4efb3fb517a140dc1a06666c135ab4fe9c7d401a7a48bf86912896414473ad4 WatchSource:0}: Error finding container e4efb3fb517a140dc1a06666c135ab4fe9c7d401a7a48bf86912896414473ad4: Status 404 returned error can't find the container with id e4efb3fb517a140dc1a06666c135ab4fe9c7d401a7a48bf86912896414473ad4 Jul 17 18:46:54.144180 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:54.144153 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s" event={"ID":"0d0c49a3-d545-4377-ac32-7b82a2df0ff1","Type":"ContainerStarted","Data":"e4efb3fb517a140dc1a06666c135ab4fe9c7d401a7a48bf86912896414473ad4"} Jul 17 18:46:54.145787 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:54.145767 2572 generic.go:356] "Generic (PLEG): container finished" podID="09ba51fc-e1fe-46ca-b42b-9c23d239782a" containerID="1c749dc588dcba92bae8c51368e8cb3aa722c68792a9b4d077261c706512bea5" exitCode=6 Jul 17 18:46:54.145887 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:54.145834 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94" event={"ID":"09ba51fc-e1fe-46ca-b42b-9c23d239782a","Type":"ContainerDied","Data":"1c749dc588dcba92bae8c51368e8cb3aa722c68792a9b4d077261c706512bea5"} Jul 17 18:46:54.145926 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:54.145891 2572 scope.go:117] "RemoveContainer" containerID="6baf3ad2b663263a074eb1ce8ae24f3d4ce131bc44f3c49a87e0c222be08ac60" Jul 17 18:46:54.146208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:54.146185 2572 scope.go:117] "RemoveContainer" containerID="1c749dc588dcba92bae8c51368e8cb3aa722c68792a9b4d077261c706512bea5" Jul 17 18:46:54.146415 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:46:54.146395 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94_odh-ai-gateway-infra(09ba51fc-e1fe-46ca-b42b-9c23d239782a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94" podUID="09ba51fc-e1fe-46ca-b42b-9c23d239782a" Jul 17 18:46:54.147718 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:54.147698 2572 generic.go:356] "Generic (PLEG): container finished" podID="4319849b-e4ac-4c5a-afbe-424ff5d72bfd" containerID="930330320a7187a03806d533f3b14019745b852f7e7f05541e0bebb98ab837fc" exitCode=6 Jul 17 18:46:54.147815 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:54.147756 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv" event={"ID":"4319849b-e4ac-4c5a-afbe-424ff5d72bfd","Type":"ContainerDied","Data":"930330320a7187a03806d533f3b14019745b852f7e7f05541e0bebb98ab837fc"} Jul 17 18:46:54.147965 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:54.147949 2572 scope.go:117] "RemoveContainer" containerID="930330320a7187a03806d533f3b14019745b852f7e7f05541e0bebb98ab837fc" Jul 17 18:46:54.148130 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:46:54.148111 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv_odh-ai-gateway-infra(4319849b-e4ac-4c5a-afbe-424ff5d72bfd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv" podUID="4319849b-e4ac-4c5a-afbe-424ff5d72bfd" Jul 17 18:46:54.161987 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:54.161971 2572 scope.go:117] "RemoveContainer" containerID="9200a9751953ed646b72df9bc6c640ea1bdc9c30564e03fafc9a8c1b44a48b5d" Jul 17 18:46:55.156798 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:55.156770 2572 scope.go:117] "RemoveContainer" containerID="930330320a7187a03806d533f3b14019745b852f7e7f05541e0bebb98ab837fc" Jul 17 18:46:55.157283 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:46:55.156984 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv_odh-ai-gateway-infra(4319849b-e4ac-4c5a-afbe-424ff5d72bfd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv" podUID="4319849b-e4ac-4c5a-afbe-424ff5d72bfd" Jul 17 18:46:55.158213 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:55.158186 2572 generic.go:356] "Generic (PLEG): container finished" podID="0d0c49a3-d545-4377-ac32-7b82a2df0ff1" containerID="01ec2a04ea924a176e5c77caa5a6f430d3a221043bb8ba341b4648a8d55769b8" exitCode=6 Jul 17 18:46:55.158335 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:55.158230 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s" event={"ID":"0d0c49a3-d545-4377-ac32-7b82a2df0ff1","Type":"ContainerDied","Data":"01ec2a04ea924a176e5c77caa5a6f430d3a221043bb8ba341b4648a8d55769b8"} Jul 17 18:46:55.158401 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:55.158392 2572 scope.go:117] "RemoveContainer" containerID="01ec2a04ea924a176e5c77caa5a6f430d3a221043bb8ba341b4648a8d55769b8" Jul 17 18:46:55.180383 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:55.173847 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94"] Jul 17 18:46:55.312210 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:55.312183 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94" Jul 17 18:46:55.458733 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:55.458710 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ba51fc-e1fe-46ca-b42b-9c23d239782a-maas-api-service-ca\") pod \"09ba51fc-e1fe-46ca-b42b-9c23d239782a\" (UID: \"09ba51fc-e1fe-46ca-b42b-9c23d239782a\") " Jul 17 18:46:55.459047 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:55.459022 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ba51fc-e1fe-46ca-b42b-9c23d239782a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "09ba51fc-e1fe-46ca-b42b-9c23d239782a" (UID: "09ba51fc-e1fe-46ca-b42b-9c23d239782a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:46:55.559542 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:55.559513 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ba51fc-e1fe-46ca-b42b-9c23d239782a-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:46:56.163559 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:56.163534 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94" Jul 17 18:46:56.164055 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:56.163579 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94" event={"ID":"09ba51fc-e1fe-46ca-b42b-9c23d239782a","Type":"ContainerDied","Data":"e764400d72d5ced8f8a9f2ed67ec0d293b70ecf4d27dc0e83d3da7fcdc5e87bf"} Jul 17 18:46:56.164055 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:56.163639 2572 scope.go:117] "RemoveContainer" containerID="1c749dc588dcba92bae8c51368e8cb3aa722c68792a9b4d077261c706512bea5" Jul 17 18:46:56.165572 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:56.165547 2572 generic.go:356] "Generic (PLEG): container finished" podID="0d0c49a3-d545-4377-ac32-7b82a2df0ff1" containerID="fc72b350dea0dd7937b103fde8c1ff21acd03bc29dbc90ddfce052e4b067d009" exitCode=6 Jul 17 18:46:56.165682 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:56.165592 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s" event={"ID":"0d0c49a3-d545-4377-ac32-7b82a2df0ff1","Type":"ContainerDied","Data":"fc72b350dea0dd7937b103fde8c1ff21acd03bc29dbc90ddfce052e4b067d009"} Jul 17 18:46:56.165828 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:56.165814 2572 scope.go:117] "RemoveContainer" containerID="fc72b350dea0dd7937b103fde8c1ff21acd03bc29dbc90ddfce052e4b067d009" Jul 17 18:46:56.166071 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:46:56.166048 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-k896s_odh-ai-gateway-infra(0d0c49a3-d545-4377-ac32-7b82a2df0ff1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s" podUID="0d0c49a3-d545-4377-ac32-7b82a2df0ff1" Jul 17 18:46:56.172042 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:56.172019 2572 scope.go:117] "RemoveContainer" containerID="01ec2a04ea924a176e5c77caa5a6f430d3a221043bb8ba341b4648a8d55769b8" Jul 17 18:46:56.199968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:56.199942 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94"] Jul 17 18:46:56.203364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:56.203344 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-5zm94"] Jul 17 18:46:57.172370 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:57.172344 2572 scope.go:117] "RemoveContainer" containerID="fc72b350dea0dd7937b103fde8c1ff21acd03bc29dbc90ddfce052e4b067d009" Jul 17 18:46:57.172735 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:46:57.172511 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-k896s_odh-ai-gateway-infra(0d0c49a3-d545-4377-ac32-7b82a2df0ff1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s" podUID="0d0c49a3-d545-4377-ac32-7b82a2df0ff1" Jul 17 18:46:57.224471 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:46:57.224442 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09ba51fc-e1fe-46ca-b42b-9c23d239782a" path="/var/lib/kubelet/pods/09ba51fc-e1fe-46ca-b42b-9c23d239782a/volumes" Jul 17 18:47:05.221802 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:05.221772 2572 scope.go:117] "RemoveContainer" containerID="481df4c8a90af1e0c00c18d35e9657dc7be69876ba1bf054c8519de93e48aed5" Jul 17 18:47:06.214820 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:06.214738 2572 generic.go:356] "Generic (PLEG): container finished" podID="d9df574e-e59c-42ef-9966-5257f9697b58" containerID="ff1507ad33a007d11595d2f4a0bd78eb716e8e6ad2abac1d8ff68ca34464a557" exitCode=6 Jul 17 18:47:06.214820 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:06.214808 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr" event={"ID":"d9df574e-e59c-42ef-9966-5257f9697b58","Type":"ContainerDied","Data":"ff1507ad33a007d11595d2f4a0bd78eb716e8e6ad2abac1d8ff68ca34464a557"} Jul 17 18:47:06.215063 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:06.214870 2572 scope.go:117] "RemoveContainer" containerID="481df4c8a90af1e0c00c18d35e9657dc7be69876ba1bf054c8519de93e48aed5" Jul 17 18:47:06.215255 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:06.215240 2572 scope.go:117] "RemoveContainer" containerID="ff1507ad33a007d11595d2f4a0bd78eb716e8e6ad2abac1d8ff68ca34464a557" Jul 17 18:47:06.215451 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:47:06.215431 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr_odh-ai-gateway-infra(d9df574e-e59c-42ef-9966-5257f9697b58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr" podUID="d9df574e-e59c-42ef-9966-5257f9697b58" Jul 17 18:47:07.239728 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:07.239694 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr"] Jul 17 18:47:07.372720 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:07.372698 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr" Jul 17 18:47:07.564423 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:07.564395 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9df574e-e59c-42ef-9966-5257f9697b58-maas-api-service-ca\") pod \"d9df574e-e59c-42ef-9966-5257f9697b58\" (UID: \"d9df574e-e59c-42ef-9966-5257f9697b58\") " Jul 17 18:47:07.564768 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:07.564743 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9df574e-e59c-42ef-9966-5257f9697b58-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d9df574e-e59c-42ef-9966-5257f9697b58" (UID: "d9df574e-e59c-42ef-9966-5257f9697b58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:47:07.665593 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:07.665551 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9df574e-e59c-42ef-9966-5257f9697b58-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:47:08.219682 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:08.219651 2572 scope.go:117] "RemoveContainer" containerID="930330320a7187a03806d533f3b14019745b852f7e7f05541e0bebb98ab837fc" Jul 17 18:47:08.225976 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:08.225957 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr" Jul 17 18:47:08.226085 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:08.225973 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr" event={"ID":"d9df574e-e59c-42ef-9966-5257f9697b58","Type":"ContainerDied","Data":"5b1aa8fed766e9303fa503905c63ae150d3274363a115f14aee3d6ec4d500569"} Jul 17 18:47:08.226085 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:08.226043 2572 scope.go:117] "RemoveContainer" containerID="ff1507ad33a007d11595d2f4a0bd78eb716e8e6ad2abac1d8ff68ca34464a557" Jul 17 18:47:08.260897 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:08.260868 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr"] Jul 17 18:47:08.264754 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:08.264730 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9xfkr"] Jul 17 18:47:09.224768 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:09.224690 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9df574e-e59c-42ef-9966-5257f9697b58" path="/var/lib/kubelet/pods/d9df574e-e59c-42ef-9966-5257f9697b58/volumes" Jul 17 18:47:09.232202 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:09.232178 2572 generic.go:356] "Generic (PLEG): container finished" podID="4319849b-e4ac-4c5a-afbe-424ff5d72bfd" containerID="a99cae298fbec0c88eeda750376699bd17165d62eb882261ec879bd19ce2ee72" exitCode=6 Jul 17 18:47:09.232353 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:09.232252 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv" event={"ID":"4319849b-e4ac-4c5a-afbe-424ff5d72bfd","Type":"ContainerDied","Data":"a99cae298fbec0c88eeda750376699bd17165d62eb882261ec879bd19ce2ee72"} Jul 17 18:47:09.232353 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:09.232308 2572 scope.go:117] "RemoveContainer" containerID="930330320a7187a03806d533f3b14019745b852f7e7f05541e0bebb98ab837fc" Jul 17 18:47:09.232648 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:09.232633 2572 scope.go:117] "RemoveContainer" containerID="a99cae298fbec0c88eeda750376699bd17165d62eb882261ec879bd19ce2ee72" Jul 17 18:47:09.232899 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:47:09.232875 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv_odh-ai-gateway-infra(4319849b-e4ac-4c5a-afbe-424ff5d72bfd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv" podUID="4319849b-e4ac-4c5a-afbe-424ff5d72bfd" Jul 17 18:47:10.259359 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:10.259329 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv"] Jul 17 18:47:10.390582 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:10.390556 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv" Jul 17 18:47:10.487784 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:10.487755 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4319849b-e4ac-4c5a-afbe-424ff5d72bfd-maas-api-service-ca\") pod \"4319849b-e4ac-4c5a-afbe-424ff5d72bfd\" (UID: \"4319849b-e4ac-4c5a-afbe-424ff5d72bfd\") " Jul 17 18:47:10.488136 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:10.488114 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4319849b-e4ac-4c5a-afbe-424ff5d72bfd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4319849b-e4ac-4c5a-afbe-424ff5d72bfd" (UID: "4319849b-e4ac-4c5a-afbe-424ff5d72bfd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:47:10.588374 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:10.588343 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4319849b-e4ac-4c5a-afbe-424ff5d72bfd-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:47:11.219642 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:11.219614 2572 scope.go:117] "RemoveContainer" containerID="fc72b350dea0dd7937b103fde8c1ff21acd03bc29dbc90ddfce052e4b067d009" Jul 17 18:47:11.244577 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:11.244560 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv" Jul 17 18:47:11.244677 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:11.244593 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv" event={"ID":"4319849b-e4ac-4c5a-afbe-424ff5d72bfd","Type":"ContainerDied","Data":"510a658df56a5818906e493a1094f2a6da1fbd3b336a0ffe8cfc33a682348387"} Jul 17 18:47:11.244677 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:11.244645 2572 scope.go:117] "RemoveContainer" containerID="a99cae298fbec0c88eeda750376699bd17165d62eb882261ec879bd19ce2ee72" Jul 17 18:47:11.266003 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:11.265979 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv"] Jul 17 18:47:11.273517 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:11.273490 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7tgsv"] Jul 17 18:47:12.251400 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:12.251316 2572 generic.go:356] "Generic (PLEG): container finished" podID="0d0c49a3-d545-4377-ac32-7b82a2df0ff1" containerID="8cfdf39b068faa73d635f710049f8b89f0541dd74a15261f258abe110f4d9d3d" exitCode=6 Jul 17 18:47:12.251400 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:12.251375 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s" event={"ID":"0d0c49a3-d545-4377-ac32-7b82a2df0ff1","Type":"ContainerDied","Data":"8cfdf39b068faa73d635f710049f8b89f0541dd74a15261f258abe110f4d9d3d"} Jul 17 18:47:12.251646 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:12.251447 2572 scope.go:117] "RemoveContainer" containerID="fc72b350dea0dd7937b103fde8c1ff21acd03bc29dbc90ddfce052e4b067d009" Jul 17 18:47:12.251784 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:12.251765 2572 scope.go:117] "RemoveContainer" containerID="8cfdf39b068faa73d635f710049f8b89f0541dd74a15261f258abe110f4d9d3d" Jul 17 18:47:12.251979 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:47:12.251960 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-k896s_odh-ai-gateway-infra(0d0c49a3-d545-4377-ac32-7b82a2df0ff1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s" podUID="0d0c49a3-d545-4377-ac32-7b82a2df0ff1" Jul 17 18:47:13.225156 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:13.225126 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4319849b-e4ac-4c5a-afbe-424ff5d72bfd" path="/var/lib/kubelet/pods/4319849b-e4ac-4c5a-afbe-424ff5d72bfd/volumes" Jul 17 18:47:13.277228 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:13.277193 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s"] Jul 17 18:47:13.405689 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:13.405658 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s" Jul 17 18:47:13.505932 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:13.505841 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d0c49a3-d545-4377-ac32-7b82a2df0ff1-maas-api-service-ca\") pod \"0d0c49a3-d545-4377-ac32-7b82a2df0ff1\" (UID: \"0d0c49a3-d545-4377-ac32-7b82a2df0ff1\") " Jul 17 18:47:13.506255 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:13.506234 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d0c49a3-d545-4377-ac32-7b82a2df0ff1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0d0c49a3-d545-4377-ac32-7b82a2df0ff1" (UID: "0d0c49a3-d545-4377-ac32-7b82a2df0ff1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:47:13.607192 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:13.607156 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d0c49a3-d545-4377-ac32-7b82a2df0ff1-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:47:14.262939 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:14.262912 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s" Jul 17 18:47:14.262939 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:14.262919 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s" event={"ID":"0d0c49a3-d545-4377-ac32-7b82a2df0ff1","Type":"ContainerDied","Data":"e4efb3fb517a140dc1a06666c135ab4fe9c7d401a7a48bf86912896414473ad4"} Jul 17 18:47:14.263496 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:14.262978 2572 scope.go:117] "RemoveContainer" containerID="8cfdf39b068faa73d635f710049f8b89f0541dd74a15261f258abe110f4d9d3d" Jul 17 18:47:14.283852 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:14.283827 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s"] Jul 17 18:47:14.289568 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:14.289540 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k896s"] Jul 17 18:47:15.224553 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:15.224523 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d0c49a3-d545-4377-ac32-7b82a2df0ff1" path="/var/lib/kubelet/pods/0d0c49a3-d545-4377-ac32-7b82a2df0ff1/volumes" Jul 17 18:47:33.441580 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441545 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pjkcm"] Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441819 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4319849b-e4ac-4c5a-afbe-424ff5d72bfd" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441830 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4319849b-e4ac-4c5a-afbe-424ff5d72bfd" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441838 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ba51fc-e1fe-46ca-b42b-9c23d239782a" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441843 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ba51fc-e1fe-46ca-b42b-9c23d239782a" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441850 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d0c49a3-d545-4377-ac32-7b82a2df0ff1" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441857 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d0c49a3-d545-4377-ac32-7b82a2df0ff1" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441862 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4319849b-e4ac-4c5a-afbe-424ff5d72bfd" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441867 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4319849b-e4ac-4c5a-afbe-424ff5d72bfd" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441879 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ba51fc-e1fe-46ca-b42b-9c23d239782a" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441884 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ba51fc-e1fe-46ca-b42b-9c23d239782a" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441890 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9df574e-e59c-42ef-9966-5257f9697b58" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441895 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9df574e-e59c-42ef-9966-5257f9697b58" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441902 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d0c49a3-d545-4377-ac32-7b82a2df0ff1" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441907 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d0c49a3-d545-4377-ac32-7b82a2df0ff1" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441913 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4319849b-e4ac-4c5a-afbe-424ff5d72bfd" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441919 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4319849b-e4ac-4c5a-afbe-424ff5d72bfd" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441928 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d0c49a3-d545-4377-ac32-7b82a2df0ff1" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441933 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d0c49a3-d545-4377-ac32-7b82a2df0ff1" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441940 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9df574e-e59c-42ef-9966-5257f9697b58" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441945 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9df574e-e59c-42ef-9966-5257f9697b58" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441951 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9df574e-e59c-42ef-9966-5257f9697b58" containerName="revoke-keys" Jul 17 18:47:33.441968 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.441956 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9df574e-e59c-42ef-9966-5257f9697b58" containerName="revoke-keys" Jul 17 18:47:33.442651 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.442036 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d0c49a3-d545-4377-ac32-7b82a2df0ff1" containerName="revoke-keys" Jul 17 18:47:33.442651 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.442044 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ba51fc-e1fe-46ca-b42b-9c23d239782a" containerName="revoke-keys" Jul 17 18:47:33.442651 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.442050 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4319849b-e4ac-4c5a-afbe-424ff5d72bfd" containerName="revoke-keys" Jul 17 18:47:33.442651 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.442056 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ba51fc-e1fe-46ca-b42b-9c23d239782a" containerName="revoke-keys" Jul 17 18:47:33.442651 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.442061 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9df574e-e59c-42ef-9966-5257f9697b58" containerName="revoke-keys" Jul 17 18:47:33.442651 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.442066 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9df574e-e59c-42ef-9966-5257f9697b58" containerName="revoke-keys" Jul 17 18:47:33.442651 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.442070 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9df574e-e59c-42ef-9966-5257f9697b58" containerName="revoke-keys" Jul 17 18:47:33.442651 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.442075 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4319849b-e4ac-4c5a-afbe-424ff5d72bfd" containerName="revoke-keys" Jul 17 18:47:33.442651 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.442080 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4319849b-e4ac-4c5a-afbe-424ff5d72bfd" containerName="revoke-keys" Jul 17 18:47:33.442651 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.442084 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d0c49a3-d545-4377-ac32-7b82a2df0ff1" containerName="revoke-keys" Jul 17 18:47:33.444897 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.444882 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pjkcm" Jul 17 18:47:33.447792 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.447768 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 18:47:33.448897 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.448876 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-6xwrw\"" Jul 17 18:47:33.449044 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.448883 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 18:47:33.453221 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.453198 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pjkcm"] Jul 17 18:47:33.577061 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.576990 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdm9t\" (UniqueName: \"kubernetes.io/projected/5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5-kube-api-access-rdm9t\") pod \"kuadrant-operator-catalog-pjkcm\" (UID: \"5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5\") " pod="kuadrant-system/kuadrant-operator-catalog-pjkcm" Jul 17 18:47:33.677435 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.677401 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdm9t\" (UniqueName: \"kubernetes.io/projected/5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5-kube-api-access-rdm9t\") pod \"kuadrant-operator-catalog-pjkcm\" (UID: \"5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5\") " pod="kuadrant-system/kuadrant-operator-catalog-pjkcm" Jul 17 18:47:33.686912 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.686878 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdm9t\" (UniqueName: \"kubernetes.io/projected/5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5-kube-api-access-rdm9t\") pod \"kuadrant-operator-catalog-pjkcm\" (UID: \"5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5\") " pod="kuadrant-system/kuadrant-operator-catalog-pjkcm" Jul 17 18:47:33.754865 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.754779 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pjkcm" Jul 17 18:47:33.903529 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.903502 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pjkcm"] Jul 17 18:47:33.906048 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:47:33.906001 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3fbacb_65f3_4e63_bfcc_7d83ee5ce9b5.slice/crio-9eabff726450127db7f586d2ef291dcba0da1237cc9debf89ea4192f437fd25b WatchSource:0}: Error finding container 9eabff726450127db7f586d2ef291dcba0da1237cc9debf89ea4192f437fd25b: Status 404 returned error can't find the container with id 9eabff726450127db7f586d2ef291dcba0da1237cc9debf89ea4192f437fd25b Jul 17 18:47:33.907419 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:33.907398 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:47:34.347552 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:34.347512 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pjkcm" event={"ID":"5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5","Type":"ContainerStarted","Data":"9eabff726450127db7f586d2ef291dcba0da1237cc9debf89ea4192f437fd25b"} Jul 17 18:47:35.573754 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:35.573695 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6"] Jul 17 18:47:35.574232 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:35.574066 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ba51fc-e1fe-46ca-b42b-9c23d239782a" containerName="revoke-keys" Jul 17 18:47:35.574232 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:35.574079 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ba51fc-e1fe-46ca-b42b-9c23d239782a" containerName="revoke-keys" Jul 17 18:47:35.574232 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:35.574140 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d0c49a3-d545-4377-ac32-7b82a2df0ff1" containerName="revoke-keys" Jul 17 18:47:35.574232 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:35.574154 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ba51fc-e1fe-46ca-b42b-9c23d239782a" containerName="revoke-keys" Jul 17 18:47:35.577304 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:35.577277 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6" Jul 17 18:47:35.579889 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:35.579861 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:47:35.579889 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:35.579865 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:47:35.585071 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:35.585048 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6"] Jul 17 18:47:35.695095 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:35.694989 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3984ad99-905a-422c-a445-21e3c76b8fa9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6\" (UID: \"3984ad99-905a-422c-a445-21e3c76b8fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6" Jul 17 18:47:35.795817 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:35.795784 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3984ad99-905a-422c-a445-21e3c76b8fa9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6\" (UID: \"3984ad99-905a-422c-a445-21e3c76b8fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6" Jul 17 18:47:35.796465 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:35.796443 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3984ad99-905a-422c-a445-21e3c76b8fa9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6\" (UID: \"3984ad99-905a-422c-a445-21e3c76b8fa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6" Jul 17 18:47:35.890499 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:35.890430 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6" Jul 17 18:47:36.048209 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:36.048186 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6"] Jul 17 18:47:36.049896 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:47:36.049874 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3984ad99_905a_422c_a445_21e3c76b8fa9.slice/crio-58797f02d77888699962ba3f64364f0d3289de4a269f01b243c80f7ef0ef7008 WatchSource:0}: Error finding container 58797f02d77888699962ba3f64364f0d3289de4a269f01b243c80f7ef0ef7008: Status 404 returned error can't find the container with id 58797f02d77888699962ba3f64364f0d3289de4a269f01b243c80f7ef0ef7008 Jul 17 18:47:36.359287 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:36.359244 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6" event={"ID":"3984ad99-905a-422c-a445-21e3c76b8fa9","Type":"ContainerStarted","Data":"bd2ad0afecf63b9f27b52c3701799c2a2e92cfe27104aab9455051be0800e991"} Jul 17 18:47:36.359444 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:36.359308 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6" event={"ID":"3984ad99-905a-422c-a445-21e3c76b8fa9","Type":"ContainerStarted","Data":"58797f02d77888699962ba3f64364f0d3289de4a269f01b243c80f7ef0ef7008"} Jul 17 18:47:36.360698 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:36.360673 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pjkcm" event={"ID":"5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5","Type":"ContainerStarted","Data":"02110fa333cf019b5dbc7fdba35ca46ca7b43f9ffde74045389685477b27d36d"} Jul 17 18:47:36.377587 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:36.377545 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6" podStartSLOduration=1.377531544 podStartE2EDuration="1.377531544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:47:36.376784318 +0000 UTC m=+3051.786211924" watchObservedRunningTime="2026-07-17 18:47:36.377531544 +0000 UTC m=+3051.786959141" Jul 17 18:47:36.395311 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:36.395270 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-pjkcm" podStartSLOduration=1.31652521 podStartE2EDuration="3.395255139s" totalImagesPullingTime="2.078729929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:47:36.393116741 +0000 UTC m=+3051.802544373" watchObservedRunningTime="2026-07-17 18:47:36.395255139 +0000 UTC m=+3051.804682744" Jul 17 18:47:37.366039 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:37.365985 2572 generic.go:356] "Generic (PLEG): container finished" podID="3984ad99-905a-422c-a445-21e3c76b8fa9" containerID="bd2ad0afecf63b9f27b52c3701799c2a2e92cfe27104aab9455051be0800e991" exitCode=6 Jul 17 18:47:37.366494 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:37.366080 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6" event={"ID":"3984ad99-905a-422c-a445-21e3c76b8fa9","Type":"ContainerDied","Data":"bd2ad0afecf63b9f27b52c3701799c2a2e92cfe27104aab9455051be0800e991"} Jul 17 18:47:37.366494 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:37.366242 2572 scope.go:117] "RemoveContainer" containerID="bd2ad0afecf63b9f27b52c3701799c2a2e92cfe27104aab9455051be0800e991" Jul 17 18:47:38.372444 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:38.372410 2572 generic.go:356] "Generic (PLEG): container finished" podID="3984ad99-905a-422c-a445-21e3c76b8fa9" containerID="1e5702217e5a47a994222e4fbd132898e8f60ac70d5420c6e937d456db320489" exitCode=6 Jul 17 18:47:38.372928 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:38.372497 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6" event={"ID":"3984ad99-905a-422c-a445-21e3c76b8fa9","Type":"ContainerDied","Data":"1e5702217e5a47a994222e4fbd132898e8f60ac70d5420c6e937d456db320489"} Jul 17 18:47:38.372928 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:38.372559 2572 scope.go:117] "RemoveContainer" containerID="bd2ad0afecf63b9f27b52c3701799c2a2e92cfe27104aab9455051be0800e991" Jul 17 18:47:38.372928 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:38.372777 2572 scope.go:117] "RemoveContainer" containerID="1e5702217e5a47a994222e4fbd132898e8f60ac70d5420c6e937d456db320489" Jul 17 18:47:38.373132 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:47:38.372991 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6_odh-ai-gateway-infra(3984ad99-905a-422c-a445-21e3c76b8fa9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6" podUID="3984ad99-905a-422c-a445-21e3c76b8fa9" Jul 17 18:47:39.379245 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:39.379218 2572 scope.go:117] "RemoveContainer" containerID="1e5702217e5a47a994222e4fbd132898e8f60ac70d5420c6e937d456db320489" Jul 17 18:47:39.379639 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:47:39.379392 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6_odh-ai-gateway-infra(3984ad99-905a-422c-a445-21e3c76b8fa9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6" podUID="3984ad99-905a-422c-a445-21e3c76b8fa9" Jul 17 18:47:43.755151 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:43.755110 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-pjkcm" Jul 17 18:47:43.755642 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:43.755200 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-pjkcm" Jul 17 18:47:43.777330 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:43.777306 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-pjkcm" Jul 17 18:47:44.422247 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:44.422221 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-pjkcm" Jul 17 18:47:44.456791 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:44.456763 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pjkcm"] Jul 17 18:47:46.409649 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:46.409609 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-pjkcm" podUID="5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5" containerName="registry-server" containerID="cri-o://02110fa333cf019b5dbc7fdba35ca46ca7b43f9ffde74045389685477b27d36d" gracePeriod=2 Jul 17 18:47:46.648892 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:46.648858 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pjkcm" Jul 17 18:47:46.792406 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:46.792374 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdm9t\" (UniqueName: \"kubernetes.io/projected/5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5-kube-api-access-rdm9t\") pod \"5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5\" (UID: \"5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5\") " Jul 17 18:47:46.794564 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:46.794530 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5-kube-api-access-rdm9t" (OuterVolumeSpecName: "kube-api-access-rdm9t") pod "5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5" (UID: "5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5"). InnerVolumeSpecName "kube-api-access-rdm9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:47:46.893845 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:46.893816 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdm9t\" (UniqueName: \"kubernetes.io/projected/5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5-kube-api-access-rdm9t\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:47:47.415302 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:47.415270 2572 generic.go:356] "Generic (PLEG): container finished" podID="5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5" containerID="02110fa333cf019b5dbc7fdba35ca46ca7b43f9ffde74045389685477b27d36d" exitCode=0 Jul 17 18:47:47.415741 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:47.415321 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pjkcm" Jul 17 18:47:47.415741 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:47.415321 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pjkcm" event={"ID":"5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5","Type":"ContainerDied","Data":"02110fa333cf019b5dbc7fdba35ca46ca7b43f9ffde74045389685477b27d36d"} Jul 17 18:47:47.415741 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:47.415448 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pjkcm" event={"ID":"5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5","Type":"ContainerDied","Data":"9eabff726450127db7f586d2ef291dcba0da1237cc9debf89ea4192f437fd25b"} Jul 17 18:47:47.415741 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:47.415476 2572 scope.go:117] "RemoveContainer" containerID="02110fa333cf019b5dbc7fdba35ca46ca7b43f9ffde74045389685477b27d36d" Jul 17 18:47:47.424703 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:47.424682 2572 scope.go:117] "RemoveContainer" containerID="02110fa333cf019b5dbc7fdba35ca46ca7b43f9ffde74045389685477b27d36d" Jul 17 18:47:47.425051 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:47:47.425027 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02110fa333cf019b5dbc7fdba35ca46ca7b43f9ffde74045389685477b27d36d\": container with ID starting with 02110fa333cf019b5dbc7fdba35ca46ca7b43f9ffde74045389685477b27d36d not found: ID does not exist" containerID="02110fa333cf019b5dbc7fdba35ca46ca7b43f9ffde74045389685477b27d36d" Jul 17 18:47:47.425108 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:47.425064 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02110fa333cf019b5dbc7fdba35ca46ca7b43f9ffde74045389685477b27d36d"} err="failed to get container status \"02110fa333cf019b5dbc7fdba35ca46ca7b43f9ffde74045389685477b27d36d\": rpc error: code = NotFound desc = could not find container \"02110fa333cf019b5dbc7fdba35ca46ca7b43f9ffde74045389685477b27d36d\": container with ID starting with 02110fa333cf019b5dbc7fdba35ca46ca7b43f9ffde74045389685477b27d36d not found: ID does not exist" Jul 17 18:47:47.432798 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:47.432770 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pjkcm"] Jul 17 18:47:47.436608 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:47.436580 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pjkcm"] Jul 17 18:47:47.704025 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:47.703974 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth"] Jul 17 18:47:47.704302 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:47.704287 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5" containerName="registry-server" Jul 17 18:47:47.704348 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:47.704304 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5" containerName="registry-server" Jul 17 18:47:47.704401 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:47.704392 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5" containerName="registry-server" Jul 17 18:47:47.710807 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:47.709643 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth" Jul 17 18:47:47.714477 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:47.714449 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth"] Jul 17 18:47:47.799639 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:47.799605 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d44f60-7906-400f-b7b7-332d4205ce5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth\" (UID: \"e5d44f60-7906-400f-b7b7-332d4205ce5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth" Jul 17 18:47:47.900658 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:47.900629 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d44f60-7906-400f-b7b7-332d4205ce5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth\" (UID: \"e5d44f60-7906-400f-b7b7-332d4205ce5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth" Jul 17 18:47:47.901280 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:47.901260 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d44f60-7906-400f-b7b7-332d4205ce5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth\" (UID: \"e5d44f60-7906-400f-b7b7-332d4205ce5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth" Jul 17 18:47:48.021511 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:48.021442 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth" Jul 17 18:47:48.142609 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:48.142586 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth"] Jul 17 18:47:48.145035 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:47:48.144992 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d44f60_7906_400f_b7b7_332d4205ce5b.slice/crio-af35e1f6e914557f5414ae93bff2dbd1117108bab76d93bbb9cc3f72736b5b77 WatchSource:0}: Error finding container af35e1f6e914557f5414ae93bff2dbd1117108bab76d93bbb9cc3f72736b5b77: Status 404 returned error can't find the container with id af35e1f6e914557f5414ae93bff2dbd1117108bab76d93bbb9cc3f72736b5b77 Jul 17 18:47:48.421402 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:48.421368 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth" event={"ID":"e5d44f60-7906-400f-b7b7-332d4205ce5b","Type":"ContainerStarted","Data":"af35e1f6e914557f5414ae93bff2dbd1117108bab76d93bbb9cc3f72736b5b77"} Jul 17 18:47:49.224670 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:49.224578 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5" path="/var/lib/kubelet/pods/5a3fbacb-65f3-4e63-bfcc-7d83ee5ce9b5/volumes" Jul 17 18:47:49.427513 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:49.427473 2572 generic.go:356] "Generic (PLEG): container finished" podID="e5d44f60-7906-400f-b7b7-332d4205ce5b" containerID="17e68157e4322e7f1da30a436c2c9d5c429c1ee0d719a2eb6a85e6229a85dac9" exitCode=6 Jul 17 18:47:49.428000 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:49.427533 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth" event={"ID":"e5d44f60-7906-400f-b7b7-332d4205ce5b","Type":"ContainerDied","Data":"17e68157e4322e7f1da30a436c2c9d5c429c1ee0d719a2eb6a85e6229a85dac9"} Jul 17 18:47:49.428000 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:49.427819 2572 scope.go:117] "RemoveContainer" containerID="17e68157e4322e7f1da30a436c2c9d5c429c1ee0d719a2eb6a85e6229a85dac9" Jul 17 18:47:50.434606 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:50.434513 2572 generic.go:356] "Generic (PLEG): container finished" podID="e5d44f60-7906-400f-b7b7-332d4205ce5b" containerID="b484c89cf5d2d139846f33a2e7710290421f2fe1306b57fcc340d7301aa2a2fe" exitCode=6 Jul 17 18:47:50.434606 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:50.434593 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth" event={"ID":"e5d44f60-7906-400f-b7b7-332d4205ce5b","Type":"ContainerDied","Data":"b484c89cf5d2d139846f33a2e7710290421f2fe1306b57fcc340d7301aa2a2fe"} Jul 17 18:47:50.435106 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:50.434654 2572 scope.go:117] "RemoveContainer" containerID="17e68157e4322e7f1da30a436c2c9d5c429c1ee0d719a2eb6a85e6229a85dac9" Jul 17 18:47:50.435106 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:50.434900 2572 scope.go:117] "RemoveContainer" containerID="b484c89cf5d2d139846f33a2e7710290421f2fe1306b57fcc340d7301aa2a2fe" Jul 17 18:47:50.435188 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:47:50.435134 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth_odh-ai-gateway-infra(e5d44f60-7906-400f-b7b7-332d4205ce5b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth" podUID="e5d44f60-7906-400f-b7b7-332d4205ce5b" Jul 17 18:47:51.182603 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:51.182568 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw"] Jul 17 18:47:51.186966 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:51.186950 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw" Jul 17 18:47:51.193360 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:51.193335 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw"] Jul 17 18:47:51.329810 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:51.329783 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5affe4e0-345b-41a5-99db-7f5cd2b5aa73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw\" (UID: \"5affe4e0-345b-41a5-99db-7f5cd2b5aa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw" Jul 17 18:47:51.430207 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:51.430178 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5affe4e0-345b-41a5-99db-7f5cd2b5aa73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw\" (UID: \"5affe4e0-345b-41a5-99db-7f5cd2b5aa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw" Jul 17 18:47:51.430737 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:51.430718 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5affe4e0-345b-41a5-99db-7f5cd2b5aa73-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw\" (UID: \"5affe4e0-345b-41a5-99db-7f5cd2b5aa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw" Jul 17 18:47:51.440309 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:51.440251 2572 scope.go:117] "RemoveContainer" containerID="b484c89cf5d2d139846f33a2e7710290421f2fe1306b57fcc340d7301aa2a2fe" Jul 17 18:47:51.440639 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:47:51.440406 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth_odh-ai-gateway-infra(e5d44f60-7906-400f-b7b7-332d4205ce5b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth" podUID="e5d44f60-7906-400f-b7b7-332d4205ce5b" Jul 17 18:47:51.498983 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:51.498957 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw" Jul 17 18:47:51.622263 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:51.622238 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw"] Jul 17 18:47:51.623990 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:47:51.623965 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5affe4e0_345b_41a5_99db_7f5cd2b5aa73.slice/crio-3c235750dc937a25b4a4fad46281bbd83db4aaf23ccfaa541d18e296582c11a2 WatchSource:0}: Error finding container 3c235750dc937a25b4a4fad46281bbd83db4aaf23ccfaa541d18e296582c11a2: Status 404 returned error can't find the container with id 3c235750dc937a25b4a4fad46281bbd83db4aaf23ccfaa541d18e296582c11a2 Jul 17 18:47:52.219564 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:52.219488 2572 scope.go:117] "RemoveContainer" containerID="1e5702217e5a47a994222e4fbd132898e8f60ac70d5420c6e937d456db320489" Jul 17 18:47:52.451705 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:52.451677 2572 generic.go:356] "Generic (PLEG): container finished" podID="5affe4e0-345b-41a5-99db-7f5cd2b5aa73" containerID="c683d0b855f4e422fd1ba4129f11615a822fab858074a87650ee5f82bb2ee402" exitCode=6 Jul 17 18:47:52.452057 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:52.451745 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw" event={"ID":"5affe4e0-345b-41a5-99db-7f5cd2b5aa73","Type":"ContainerDied","Data":"c683d0b855f4e422fd1ba4129f11615a822fab858074a87650ee5f82bb2ee402"} Jul 17 18:47:52.452057 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:52.451805 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw" event={"ID":"5affe4e0-345b-41a5-99db-7f5cd2b5aa73","Type":"ContainerStarted","Data":"3c235750dc937a25b4a4fad46281bbd83db4aaf23ccfaa541d18e296582c11a2"} Jul 17 18:47:52.452057 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:52.451904 2572 scope.go:117] "RemoveContainer" containerID="c683d0b855f4e422fd1ba4129f11615a822fab858074a87650ee5f82bb2ee402" Jul 17 18:47:53.457620 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:53.457588 2572 generic.go:356] "Generic (PLEG): container finished" podID="5affe4e0-345b-41a5-99db-7f5cd2b5aa73" containerID="f1a89107a582c956ae237592e766d274111df2c9a170bca30c07280ae72e156d" exitCode=6 Jul 17 18:47:53.458084 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:53.457641 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw" event={"ID":"5affe4e0-345b-41a5-99db-7f5cd2b5aa73","Type":"ContainerDied","Data":"f1a89107a582c956ae237592e766d274111df2c9a170bca30c07280ae72e156d"} Jul 17 18:47:53.458084 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:53.457701 2572 scope.go:117] "RemoveContainer" containerID="c683d0b855f4e422fd1ba4129f11615a822fab858074a87650ee5f82bb2ee402" Jul 17 18:47:53.458084 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:53.457866 2572 scope.go:117] "RemoveContainer" containerID="f1a89107a582c956ae237592e766d274111df2c9a170bca30c07280ae72e156d" Jul 17 18:47:53.458235 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:47:53.458106 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw_odh-ai-gateway-infra(5affe4e0-345b-41a5-99db-7f5cd2b5aa73)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw" podUID="5affe4e0-345b-41a5-99db-7f5cd2b5aa73" Jul 17 18:47:53.459614 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:53.459593 2572 generic.go:356] "Generic (PLEG): container finished" podID="3984ad99-905a-422c-a445-21e3c76b8fa9" containerID="89fb9c86b60a44f450f9053965af12e5491666b8b0774f27d764c6b32d2c2b2d" exitCode=6 Jul 17 18:47:53.459709 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:53.459633 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6" event={"ID":"3984ad99-905a-422c-a445-21e3c76b8fa9","Type":"ContainerDied","Data":"89fb9c86b60a44f450f9053965af12e5491666b8b0774f27d764c6b32d2c2b2d"} Jul 17 18:47:53.459882 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:53.459868 2572 scope.go:117] "RemoveContainer" containerID="89fb9c86b60a44f450f9053965af12e5491666b8b0774f27d764c6b32d2c2b2d" Jul 17 18:47:53.460065 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:47:53.460049 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6_odh-ai-gateway-infra(3984ad99-905a-422c-a445-21e3c76b8fa9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6" podUID="3984ad99-905a-422c-a445-21e3c76b8fa9" Jul 17 18:47:53.467778 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:53.467762 2572 scope.go:117] "RemoveContainer" containerID="1e5702217e5a47a994222e4fbd132898e8f60ac70d5420c6e937d456db320489" Jul 17 18:47:53.733961 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:53.733885 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr"] Jul 17 18:47:53.738405 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:53.738386 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr" Jul 17 18:47:53.744312 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:53.744288 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr"] Jul 17 18:47:53.853292 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:53.853256 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08ca1e6-1fdf-4722-beb7-e94cebd3cae6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr\" (UID: \"f08ca1e6-1fdf-4722-beb7-e94cebd3cae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr" Jul 17 18:47:53.954429 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:53.954396 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08ca1e6-1fdf-4722-beb7-e94cebd3cae6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr\" (UID: \"f08ca1e6-1fdf-4722-beb7-e94cebd3cae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr" Jul 17 18:47:53.955001 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:53.954983 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08ca1e6-1fdf-4722-beb7-e94cebd3cae6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr\" (UID: \"f08ca1e6-1fdf-4722-beb7-e94cebd3cae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr" Jul 17 18:47:54.050546 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:54.050522 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr" Jul 17 18:47:54.174053 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:54.174020 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr"] Jul 17 18:47:54.176841 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:47:54.176816 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08ca1e6_1fdf_4722_beb7_e94cebd3cae6.slice/crio-cc19692989df2a93de44e6619a0d5c2aa572caa2cd7f12b21914c5c98c6b5b4a WatchSource:0}: Error finding container cc19692989df2a93de44e6619a0d5c2aa572caa2cd7f12b21914c5c98c6b5b4a: Status 404 returned error can't find the container with id cc19692989df2a93de44e6619a0d5c2aa572caa2cd7f12b21914c5c98c6b5b4a Jul 17 18:47:54.465577 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:54.465549 2572 scope.go:117] "RemoveContainer" containerID="f1a89107a582c956ae237592e766d274111df2c9a170bca30c07280ae72e156d" Jul 17 18:47:54.466000 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:47:54.465789 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw_odh-ai-gateway-infra(5affe4e0-345b-41a5-99db-7f5cd2b5aa73)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw" podUID="5affe4e0-345b-41a5-99db-7f5cd2b5aa73" Jul 17 18:47:54.466844 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:54.466816 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr" event={"ID":"f08ca1e6-1fdf-4722-beb7-e94cebd3cae6","Type":"ContainerStarted","Data":"07f17449df4b455e4c5e467a031195a51f9eb061b8c1e1f85f45bb51278680b9"} Jul 17 18:47:54.466974 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:54.466872 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr" event={"ID":"f08ca1e6-1fdf-4722-beb7-e94cebd3cae6","Type":"ContainerStarted","Data":"cc19692989df2a93de44e6619a0d5c2aa572caa2cd7f12b21914c5c98c6b5b4a"} Jul 17 18:47:54.494657 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:54.494637 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6"] Jul 17 18:47:54.497730 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:54.497687 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr" podStartSLOduration=1.4976730919999999 podStartE2EDuration="1.497673092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:47:54.495571351 +0000 UTC m=+3069.904998976" watchObservedRunningTime="2026-07-17 18:47:54.497673092 +0000 UTC m=+3069.907100697" Jul 17 18:47:54.613774 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:54.613752 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6" Jul 17 18:47:54.762373 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:54.762352 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3984ad99-905a-422c-a445-21e3c76b8fa9-maas-api-service-ca\") pod \"3984ad99-905a-422c-a445-21e3c76b8fa9\" (UID: \"3984ad99-905a-422c-a445-21e3c76b8fa9\") " Jul 17 18:47:54.762685 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:54.762660 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3984ad99-905a-422c-a445-21e3c76b8fa9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3984ad99-905a-422c-a445-21e3c76b8fa9" (UID: "3984ad99-905a-422c-a445-21e3c76b8fa9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:47:54.863893 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:54.863813 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3984ad99-905a-422c-a445-21e3c76b8fa9-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:47:55.473789 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:55.473762 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6" Jul 17 18:47:55.474239 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:55.473785 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6" event={"ID":"3984ad99-905a-422c-a445-21e3c76b8fa9","Type":"ContainerDied","Data":"58797f02d77888699962ba3f64364f0d3289de4a269f01b243c80f7ef0ef7008"} Jul 17 18:47:55.474239 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:55.473855 2572 scope.go:117] "RemoveContainer" containerID="89fb9c86b60a44f450f9053965af12e5491666b8b0774f27d764c6b32d2c2b2d" Jul 17 18:47:55.479056 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:55.479032 2572 generic.go:356] "Generic (PLEG): container finished" podID="f08ca1e6-1fdf-4722-beb7-e94cebd3cae6" containerID="07f17449df4b455e4c5e467a031195a51f9eb061b8c1e1f85f45bb51278680b9" exitCode=6 Jul 17 18:47:55.479179 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:55.479130 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr" event={"ID":"f08ca1e6-1fdf-4722-beb7-e94cebd3cae6","Type":"ContainerDied","Data":"07f17449df4b455e4c5e467a031195a51f9eb061b8c1e1f85f45bb51278680b9"} Jul 17 18:47:55.479274 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:55.479260 2572 scope.go:117] "RemoveContainer" containerID="07f17449df4b455e4c5e467a031195a51f9eb061b8c1e1f85f45bb51278680b9" Jul 17 18:47:55.491285 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:55.491241 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6"] Jul 17 18:47:55.493344 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:55.493322 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-slsf6"] Jul 17 18:47:56.484848 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:56.484755 2572 generic.go:356] "Generic (PLEG): container finished" podID="f08ca1e6-1fdf-4722-beb7-e94cebd3cae6" containerID="a839cd304fbd25989f1a6efd8f2c6b2570e91ee2dd8c35328ffcbbd8e4137355" exitCode=6 Jul 17 18:47:56.485334 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:56.484839 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr" event={"ID":"f08ca1e6-1fdf-4722-beb7-e94cebd3cae6","Type":"ContainerDied","Data":"a839cd304fbd25989f1a6efd8f2c6b2570e91ee2dd8c35328ffcbbd8e4137355"} Jul 17 18:47:56.485334 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:56.484908 2572 scope.go:117] "RemoveContainer" containerID="07f17449df4b455e4c5e467a031195a51f9eb061b8c1e1f85f45bb51278680b9" Jul 17 18:47:56.485334 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:56.485126 2572 scope.go:117] "RemoveContainer" containerID="a839cd304fbd25989f1a6efd8f2c6b2570e91ee2dd8c35328ffcbbd8e4137355" Jul 17 18:47:56.485510 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:47:56.485357 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr_odh-ai-gateway-infra(f08ca1e6-1fdf-4722-beb7-e94cebd3cae6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr" podUID="f08ca1e6-1fdf-4722-beb7-e94cebd3cae6" Jul 17 18:47:57.225990 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:57.225954 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3984ad99-905a-422c-a445-21e3c76b8fa9" path="/var/lib/kubelet/pods/3984ad99-905a-422c-a445-21e3c76b8fa9/volumes" Jul 17 18:47:57.493292 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:47:57.493213 2572 scope.go:117] "RemoveContainer" containerID="a839cd304fbd25989f1a6efd8f2c6b2570e91ee2dd8c35328ffcbbd8e4137355" Jul 17 18:47:57.493669 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:47:57.493380 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr_odh-ai-gateway-infra(f08ca1e6-1fdf-4722-beb7-e94cebd3cae6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr" podUID="f08ca1e6-1fdf-4722-beb7-e94cebd3cae6" Jul 17 18:48:02.220206 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:02.220171 2572 scope.go:117] "RemoveContainer" containerID="b484c89cf5d2d139846f33a2e7710290421f2fe1306b57fcc340d7301aa2a2fe" Jul 17 18:48:02.516665 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:02.516596 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth" event={"ID":"e5d44f60-7906-400f-b7b7-332d4205ce5b","Type":"ContainerStarted","Data":"8946c60aeb4ddda0cd1c0cbcb5ea7a17c7a45376dfe3de7d3c27b5dc20daa5b0"} Jul 17 18:48:02.534191 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:02.534153 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth" podStartSLOduration=15.534115495 podStartE2EDuration="15.534115495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:48:02.53349791 +0000 UTC m=+3077.942925517" watchObservedRunningTime="2026-07-17 18:48:02.534115495 +0000 UTC m=+3077.943543560" Jul 17 18:48:03.522403 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:03.522367 2572 generic.go:356] "Generic (PLEG): container finished" podID="e5d44f60-7906-400f-b7b7-332d4205ce5b" containerID="8946c60aeb4ddda0cd1c0cbcb5ea7a17c7a45376dfe3de7d3c27b5dc20daa5b0" exitCode=6 Jul 17 18:48:03.522829 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:03.522437 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth" event={"ID":"e5d44f60-7906-400f-b7b7-332d4205ce5b","Type":"ContainerDied","Data":"8946c60aeb4ddda0cd1c0cbcb5ea7a17c7a45376dfe3de7d3c27b5dc20daa5b0"} Jul 17 18:48:03.522829 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:03.522497 2572 scope.go:117] "RemoveContainer" containerID="b484c89cf5d2d139846f33a2e7710290421f2fe1306b57fcc340d7301aa2a2fe" Jul 17 18:48:03.522829 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:03.522748 2572 scope.go:117] "RemoveContainer" containerID="8946c60aeb4ddda0cd1c0cbcb5ea7a17c7a45376dfe3de7d3c27b5dc20daa5b0" Jul 17 18:48:03.523004 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:48:03.522953 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth_odh-ai-gateway-infra(e5d44f60-7906-400f-b7b7-332d4205ce5b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth" podUID="e5d44f60-7906-400f-b7b7-332d4205ce5b" Jul 17 18:48:03.542956 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:03.542932 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth"] Jul 17 18:48:04.656530 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:04.656509 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth" Jul 17 18:48:04.740877 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:04.740850 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d44f60-7906-400f-b7b7-332d4205ce5b-maas-api-service-ca\") pod \"e5d44f60-7906-400f-b7b7-332d4205ce5b\" (UID: \"e5d44f60-7906-400f-b7b7-332d4205ce5b\") " Jul 17 18:48:04.741201 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:04.741179 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d44f60-7906-400f-b7b7-332d4205ce5b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5d44f60-7906-400f-b7b7-332d4205ce5b" (UID: "e5d44f60-7906-400f-b7b7-332d4205ce5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:48:04.842324 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:04.842243 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d44f60-7906-400f-b7b7-332d4205ce5b-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:48:05.533981 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:05.533949 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth" Jul 17 18:48:05.534154 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:05.533972 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth" event={"ID":"e5d44f60-7906-400f-b7b7-332d4205ce5b","Type":"ContainerDied","Data":"af35e1f6e914557f5414ae93bff2dbd1117108bab76d93bbb9cc3f72736b5b77"} Jul 17 18:48:05.534154 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:05.534067 2572 scope.go:117] "RemoveContainer" containerID="8946c60aeb4ddda0cd1c0cbcb5ea7a17c7a45376dfe3de7d3c27b5dc20daa5b0" Jul 17 18:48:05.550612 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:05.550587 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth"] Jul 17 18:48:05.555928 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:05.555902 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gjxth"] Jul 17 18:48:07.224219 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:07.224189 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5d44f60-7906-400f-b7b7-332d4205ce5b" path="/var/lib/kubelet/pods/e5d44f60-7906-400f-b7b7-332d4205ce5b/volumes" Jul 17 18:48:09.220620 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:09.220585 2572 scope.go:117] "RemoveContainer" containerID="f1a89107a582c956ae237592e766d274111df2c9a170bca30c07280ae72e156d" Jul 17 18:48:09.553744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:09.553712 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw" event={"ID":"5affe4e0-345b-41a5-99db-7f5cd2b5aa73","Type":"ContainerStarted","Data":"c851992da5d2fa273a9d1a0762cf0864488abb31a8910f1c745120008bedf329"} Jul 17 18:48:09.571351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:09.571304 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw" podStartSLOduration=18.57128504 podStartE2EDuration="18.57128504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:48:09.570554456 +0000 UTC m=+3084.979982082" watchObservedRunningTime="2026-07-17 18:48:09.57128504 +0000 UTC m=+3084.980712646" Jul 17 18:48:10.559649 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:10.559617 2572 generic.go:356] "Generic (PLEG): container finished" podID="5affe4e0-345b-41a5-99db-7f5cd2b5aa73" containerID="c851992da5d2fa273a9d1a0762cf0864488abb31a8910f1c745120008bedf329" exitCode=6 Jul 17 18:48:10.560070 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:10.559688 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw" event={"ID":"5affe4e0-345b-41a5-99db-7f5cd2b5aa73","Type":"ContainerDied","Data":"c851992da5d2fa273a9d1a0762cf0864488abb31a8910f1c745120008bedf329"} Jul 17 18:48:10.560070 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:10.559751 2572 scope.go:117] "RemoveContainer" containerID="f1a89107a582c956ae237592e766d274111df2c9a170bca30c07280ae72e156d" Jul 17 18:48:10.560070 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:10.560042 2572 scope.go:117] "RemoveContainer" containerID="c851992da5d2fa273a9d1a0762cf0864488abb31a8910f1c745120008bedf329" Jul 17 18:48:10.560288 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:48:10.560268 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw_odh-ai-gateway-infra(5affe4e0-345b-41a5-99db-7f5cd2b5aa73)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw" podUID="5affe4e0-345b-41a5-99db-7f5cd2b5aa73" Jul 17 18:48:10.586717 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:10.586691 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw"] Jul 17 18:48:11.219895 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:11.219861 2572 scope.go:117] "RemoveContainer" containerID="a839cd304fbd25989f1a6efd8f2c6b2570e91ee2dd8c35328ffcbbd8e4137355" Jul 17 18:48:11.567328 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:11.567297 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr" event={"ID":"f08ca1e6-1fdf-4722-beb7-e94cebd3cae6","Type":"ContainerStarted","Data":"ac8852d844a5d145617c232aae8247f75e25eb85ffc41cd998d4e01f4551b3a9"} Jul 17 18:48:11.695966 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:11.695942 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw" Jul 17 18:48:11.800668 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:11.800638 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5affe4e0-345b-41a5-99db-7f5cd2b5aa73-maas-api-service-ca\") pod \"5affe4e0-345b-41a5-99db-7f5cd2b5aa73\" (UID: \"5affe4e0-345b-41a5-99db-7f5cd2b5aa73\") " Jul 17 18:48:11.800980 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:11.800955 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5affe4e0-345b-41a5-99db-7f5cd2b5aa73-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5affe4e0-345b-41a5-99db-7f5cd2b5aa73" (UID: "5affe4e0-345b-41a5-99db-7f5cd2b5aa73"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:48:11.901436 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:11.901409 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5affe4e0-345b-41a5-99db-7f5cd2b5aa73-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:48:12.573726 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:12.573692 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw" Jul 17 18:48:12.574201 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:12.573701 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw" event={"ID":"5affe4e0-345b-41a5-99db-7f5cd2b5aa73","Type":"ContainerDied","Data":"3c235750dc937a25b4a4fad46281bbd83db4aaf23ccfaa541d18e296582c11a2"} Jul 17 18:48:12.574201 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:12.573819 2572 scope.go:117] "RemoveContainer" containerID="c851992da5d2fa273a9d1a0762cf0864488abb31a8910f1c745120008bedf329" Jul 17 18:48:12.575638 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:12.575613 2572 generic.go:356] "Generic (PLEG): container finished" podID="f08ca1e6-1fdf-4722-beb7-e94cebd3cae6" containerID="ac8852d844a5d145617c232aae8247f75e25eb85ffc41cd998d4e01f4551b3a9" exitCode=6 Jul 17 18:48:12.575718 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:12.575684 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr" event={"ID":"f08ca1e6-1fdf-4722-beb7-e94cebd3cae6","Type":"ContainerDied","Data":"ac8852d844a5d145617c232aae8247f75e25eb85ffc41cd998d4e01f4551b3a9"} Jul 17 18:48:12.576035 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:12.576002 2572 scope.go:117] "RemoveContainer" containerID="ac8852d844a5d145617c232aae8247f75e25eb85ffc41cd998d4e01f4551b3a9" Jul 17 18:48:12.576255 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:48:12.576235 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr_odh-ai-gateway-infra(f08ca1e6-1fdf-4722-beb7-e94cebd3cae6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr" podUID="f08ca1e6-1fdf-4722-beb7-e94cebd3cae6" Jul 17 18:48:12.582618 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:12.582602 2572 scope.go:117] "RemoveContainer" containerID="a839cd304fbd25989f1a6efd8f2c6b2570e91ee2dd8c35328ffcbbd8e4137355" Jul 17 18:48:12.606292 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:12.606270 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw"] Jul 17 18:48:12.609127 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:12.609106 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr"] Jul 17 18:48:12.611784 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:12.611764 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-k7lkw"] Jul 17 18:48:13.224191 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:13.224161 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5affe4e0-345b-41a5-99db-7f5cd2b5aa73" path="/var/lib/kubelet/pods/5affe4e0-345b-41a5-99db-7f5cd2b5aa73/volumes" Jul 17 18:48:13.708344 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:13.708323 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr" Jul 17 18:48:13.815607 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:13.815577 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08ca1e6-1fdf-4722-beb7-e94cebd3cae6-maas-api-service-ca\") pod \"f08ca1e6-1fdf-4722-beb7-e94cebd3cae6\" (UID: \"f08ca1e6-1fdf-4722-beb7-e94cebd3cae6\") " Jul 17 18:48:13.815935 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:13.815911 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08ca1e6-1fdf-4722-beb7-e94cebd3cae6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f08ca1e6-1fdf-4722-beb7-e94cebd3cae6" (UID: "f08ca1e6-1fdf-4722-beb7-e94cebd3cae6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:48:13.916950 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:13.916872 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08ca1e6-1fdf-4722-beb7-e94cebd3cae6-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:48:14.587611 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:14.587579 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr" event={"ID":"f08ca1e6-1fdf-4722-beb7-e94cebd3cae6","Type":"ContainerDied","Data":"cc19692989df2a93de44e6619a0d5c2aa572caa2cd7f12b21914c5c98c6b5b4a"} Jul 17 18:48:14.587800 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:14.587634 2572 scope.go:117] "RemoveContainer" containerID="ac8852d844a5d145617c232aae8247f75e25eb85ffc41cd998d4e01f4551b3a9" Jul 17 18:48:14.587800 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:14.587589 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr" Jul 17 18:48:14.608338 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:14.608314 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr"] Jul 17 18:48:14.611618 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:14.611594 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-k8gtr"] Jul 17 18:48:15.224725 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:15.224692 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f08ca1e6-1fdf-4722-beb7-e94cebd3cae6" path="/var/lib/kubelet/pods/f08ca1e6-1fdf-4722-beb7-e94cebd3cae6/volumes" Jul 17 18:48:25.613559 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.613527 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48"] Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.613948 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d44f60-7906-400f-b7b7-332d4205ce5b" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.613965 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d44f60-7906-400f-b7b7-332d4205ce5b" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.613980 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3984ad99-905a-422c-a445-21e3c76b8fa9" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.613988 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3984ad99-905a-422c-a445-21e3c76b8fa9" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.613998 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d44f60-7906-400f-b7b7-332d4205ce5b" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614028 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d44f60-7906-400f-b7b7-332d4205ce5b" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614038 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3984ad99-905a-422c-a445-21e3c76b8fa9" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614047 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3984ad99-905a-422c-a445-21e3c76b8fa9" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614057 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f08ca1e6-1fdf-4722-beb7-e94cebd3cae6" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614065 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f08ca1e6-1fdf-4722-beb7-e94cebd3cae6" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614078 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f08ca1e6-1fdf-4722-beb7-e94cebd3cae6" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614087 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f08ca1e6-1fdf-4722-beb7-e94cebd3cae6" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614098 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5affe4e0-345b-41a5-99db-7f5cd2b5aa73" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614106 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5affe4e0-345b-41a5-99db-7f5cd2b5aa73" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614118 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5affe4e0-345b-41a5-99db-7f5cd2b5aa73" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614127 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5affe4e0-345b-41a5-99db-7f5cd2b5aa73" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614136 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5affe4e0-345b-41a5-99db-7f5cd2b5aa73" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614145 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5affe4e0-345b-41a5-99db-7f5cd2b5aa73" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614167 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d44f60-7906-400f-b7b7-332d4205ce5b" containerName="revoke-keys" Jul 17 18:48:25.614183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614174 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d44f60-7906-400f-b7b7-332d4205ce5b" containerName="revoke-keys" Jul 17 18:48:25.615174 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614262 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f08ca1e6-1fdf-4722-beb7-e94cebd3cae6" containerName="revoke-keys" Jul 17 18:48:25.615174 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614274 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d44f60-7906-400f-b7b7-332d4205ce5b" containerName="revoke-keys" Jul 17 18:48:25.615174 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614286 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f08ca1e6-1fdf-4722-beb7-e94cebd3cae6" containerName="revoke-keys" Jul 17 18:48:25.615174 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614295 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5affe4e0-345b-41a5-99db-7f5cd2b5aa73" containerName="revoke-keys" Jul 17 18:48:25.615174 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614303 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3984ad99-905a-422c-a445-21e3c76b8fa9" containerName="revoke-keys" Jul 17 18:48:25.615174 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614310 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5affe4e0-345b-41a5-99db-7f5cd2b5aa73" containerName="revoke-keys" Jul 17 18:48:25.615174 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614318 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3984ad99-905a-422c-a445-21e3c76b8fa9" containerName="revoke-keys" Jul 17 18:48:25.615174 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614326 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5affe4e0-345b-41a5-99db-7f5cd2b5aa73" containerName="revoke-keys" Jul 17 18:48:25.615174 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.614333 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d44f60-7906-400f-b7b7-332d4205ce5b" containerName="revoke-keys" Jul 17 18:48:25.617461 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.617440 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48" Jul 17 18:48:25.619906 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.619888 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:48:25.620039 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.619931 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:48:25.623741 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.623719 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48"] Jul 17 18:48:25.718965 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.718933 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb10b49-3cfc-4e29-9dd6-811b4ce85429-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48\" (UID: \"ebb10b49-3cfc-4e29-9dd6-811b4ce85429\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48" Jul 17 18:48:25.820004 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.819974 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb10b49-3cfc-4e29-9dd6-811b4ce85429-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48\" (UID: \"ebb10b49-3cfc-4e29-9dd6-811b4ce85429\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48" Jul 17 18:48:25.820577 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.820560 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb10b49-3cfc-4e29-9dd6-811b4ce85429-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48\" (UID: \"ebb10b49-3cfc-4e29-9dd6-811b4ce85429\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48" Jul 17 18:48:25.930072 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:25.929978 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48" Jul 17 18:48:26.053292 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:26.053269 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48"] Jul 17 18:48:26.055448 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:48:26.055422 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb10b49_3cfc_4e29_9dd6_811b4ce85429.slice/crio-4d8372101ac692ff7588bca11aca5ede2f250ad4147e6566d2de5ea705f46ca6 WatchSource:0}: Error finding container 4d8372101ac692ff7588bca11aca5ede2f250ad4147e6566d2de5ea705f46ca6: Status 404 returned error can't find the container with id 4d8372101ac692ff7588bca11aca5ede2f250ad4147e6566d2de5ea705f46ca6 Jul 17 18:48:26.645076 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:26.645044 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48" event={"ID":"ebb10b49-3cfc-4e29-9dd6-811b4ce85429","Type":"ContainerStarted","Data":"533cd0cfe594b83dfda80371125ae899879fa14e0d899db77b79c1ed87a87b41"} Jul 17 18:48:26.645472 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:26.645100 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48" event={"ID":"ebb10b49-3cfc-4e29-9dd6-811b4ce85429","Type":"ContainerStarted","Data":"4d8372101ac692ff7588bca11aca5ede2f250ad4147e6566d2de5ea705f46ca6"} Jul 17 18:48:26.645472 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:26.645267 2572 scope.go:117] "RemoveContainer" containerID="533cd0cfe594b83dfda80371125ae899879fa14e0d899db77b79c1ed87a87b41" Jul 17 18:48:27.650839 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:27.650802 2572 generic.go:356] "Generic (PLEG): container finished" podID="ebb10b49-3cfc-4e29-9dd6-811b4ce85429" containerID="533cd0cfe594b83dfda80371125ae899879fa14e0d899db77b79c1ed87a87b41" exitCode=6 Jul 17 18:48:27.650839 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:27.650842 2572 generic.go:356] "Generic (PLEG): container finished" podID="ebb10b49-3cfc-4e29-9dd6-811b4ce85429" containerID="f8874827ce7355f88fce086a9fc93eb03b05f16b22fa9bda2300115f4c6e5f79" exitCode=6 Jul 17 18:48:27.651326 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:27.650889 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48" event={"ID":"ebb10b49-3cfc-4e29-9dd6-811b4ce85429","Type":"ContainerDied","Data":"533cd0cfe594b83dfda80371125ae899879fa14e0d899db77b79c1ed87a87b41"} Jul 17 18:48:27.651326 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:27.650950 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48" event={"ID":"ebb10b49-3cfc-4e29-9dd6-811b4ce85429","Type":"ContainerDied","Data":"f8874827ce7355f88fce086a9fc93eb03b05f16b22fa9bda2300115f4c6e5f79"} Jul 17 18:48:27.651326 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:27.650977 2572 scope.go:117] "RemoveContainer" containerID="533cd0cfe594b83dfda80371125ae899879fa14e0d899db77b79c1ed87a87b41" Jul 17 18:48:27.651326 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:27.651152 2572 scope.go:117] "RemoveContainer" containerID="f8874827ce7355f88fce086a9fc93eb03b05f16b22fa9bda2300115f4c6e5f79" Jul 17 18:48:27.651478 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:48:27.651388 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48_odh-ai-gateway-infra(ebb10b49-3cfc-4e29-9dd6-811b4ce85429)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48" podUID="ebb10b49-3cfc-4e29-9dd6-811b4ce85429" Jul 17 18:48:27.661557 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:27.661535 2572 scope.go:117] "RemoveContainer" containerID="533cd0cfe594b83dfda80371125ae899879fa14e0d899db77b79c1ed87a87b41" Jul 17 18:48:27.661813 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:48:27.661795 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533cd0cfe594b83dfda80371125ae899879fa14e0d899db77b79c1ed87a87b41\": container with ID starting with 533cd0cfe594b83dfda80371125ae899879fa14e0d899db77b79c1ed87a87b41 not found: ID does not exist" containerID="533cd0cfe594b83dfda80371125ae899879fa14e0d899db77b79c1ed87a87b41" Jul 17 18:48:27.661886 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:27.661819 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533cd0cfe594b83dfda80371125ae899879fa14e0d899db77b79c1ed87a87b41"} err="failed to get container status \"533cd0cfe594b83dfda80371125ae899879fa14e0d899db77b79c1ed87a87b41\": rpc error: code = NotFound desc = could not find container \"533cd0cfe594b83dfda80371125ae899879fa14e0d899db77b79c1ed87a87b41\": container with ID starting with 533cd0cfe594b83dfda80371125ae899879fa14e0d899db77b79c1ed87a87b41 not found: ID does not exist" Jul 17 18:48:28.657448 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:28.657421 2572 scope.go:117] "RemoveContainer" containerID="f8874827ce7355f88fce086a9fc93eb03b05f16b22fa9bda2300115f4c6e5f79" Jul 17 18:48:28.657872 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:48:28.657590 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48_odh-ai-gateway-infra(ebb10b49-3cfc-4e29-9dd6-811b4ce85429)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48" podUID="ebb10b49-3cfc-4e29-9dd6-811b4ce85429" Jul 17 18:48:29.953305 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:29.953272 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6"] Jul 17 18:48:29.953700 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:29.953587 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3984ad99-905a-422c-a445-21e3c76b8fa9" containerName="revoke-keys" Jul 17 18:48:29.953700 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:29.953599 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3984ad99-905a-422c-a445-21e3c76b8fa9" containerName="revoke-keys" Jul 17 18:48:29.953700 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:29.953609 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f08ca1e6-1fdf-4722-beb7-e94cebd3cae6" containerName="revoke-keys" Jul 17 18:48:29.953700 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:29.953614 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f08ca1e6-1fdf-4722-beb7-e94cebd3cae6" containerName="revoke-keys" Jul 17 18:48:29.953700 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:29.953663 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3984ad99-905a-422c-a445-21e3c76b8fa9" containerName="revoke-keys" Jul 17 18:48:29.953700 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:29.953674 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f08ca1e6-1fdf-4722-beb7-e94cebd3cae6" containerName="revoke-keys" Jul 17 18:48:29.953700 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:29.953680 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d44f60-7906-400f-b7b7-332d4205ce5b" containerName="revoke-keys" Jul 17 18:48:29.957686 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:29.957670 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6" Jul 17 18:48:29.963323 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:29.963298 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6"] Jul 17 18:48:30.057599 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:30.057562 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17beeeb-6770-4dda-aaf6-4789918798db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6\" (UID: \"e17beeeb-6770-4dda-aaf6-4789918798db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6" Jul 17 18:48:30.158964 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:30.158929 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17beeeb-6770-4dda-aaf6-4789918798db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6\" (UID: \"e17beeeb-6770-4dda-aaf6-4789918798db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6" Jul 17 18:48:30.159585 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:30.159566 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17beeeb-6770-4dda-aaf6-4789918798db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6\" (UID: \"e17beeeb-6770-4dda-aaf6-4789918798db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6" Jul 17 18:48:30.269814 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:30.269724 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6" Jul 17 18:48:30.600609 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:30.600585 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6"] Jul 17 18:48:30.602745 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:48:30.602716 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17beeeb_6770_4dda_aaf6_4789918798db.slice/crio-a871ef56aabc5a222395744794a80684accd408cff7f589cfa4dcda78284c070 WatchSource:0}: Error finding container a871ef56aabc5a222395744794a80684accd408cff7f589cfa4dcda78284c070: Status 404 returned error can't find the container with id a871ef56aabc5a222395744794a80684accd408cff7f589cfa4dcda78284c070 Jul 17 18:48:30.667573 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:30.667550 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6" event={"ID":"e17beeeb-6770-4dda-aaf6-4789918798db","Type":"ContainerStarted","Data":"a871ef56aabc5a222395744794a80684accd408cff7f589cfa4dcda78284c070"} Jul 17 18:48:31.673796 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:31.673758 2572 generic.go:356] "Generic (PLEG): container finished" podID="e17beeeb-6770-4dda-aaf6-4789918798db" containerID="b1f512b956138f98920fa298c2f75dd490d72c42000ca05357d0f926fa4a980d" exitCode=7 Jul 17 18:48:31.674186 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:31.673835 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6" event={"ID":"e17beeeb-6770-4dda-aaf6-4789918798db","Type":"ContainerDied","Data":"b1f512b956138f98920fa298c2f75dd490d72c42000ca05357d0f926fa4a980d"} Jul 17 18:48:31.674186 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:31.674037 2572 scope.go:117] "RemoveContainer" containerID="b1f512b956138f98920fa298c2f75dd490d72c42000ca05357d0f926fa4a980d" Jul 17 18:48:32.680324 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:32.680287 2572 generic.go:356] "Generic (PLEG): container finished" podID="e17beeeb-6770-4dda-aaf6-4789918798db" containerID="ab943ffa97dfe9a0a17ea7c16c305ff1427c10d813f23265b538fc70d6e88c6c" exitCode=7 Jul 17 18:48:32.680773 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:32.680378 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6" event={"ID":"e17beeeb-6770-4dda-aaf6-4789918798db","Type":"ContainerDied","Data":"ab943ffa97dfe9a0a17ea7c16c305ff1427c10d813f23265b538fc70d6e88c6c"} Jul 17 18:48:32.680773 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:32.680446 2572 scope.go:117] "RemoveContainer" containerID="b1f512b956138f98920fa298c2f75dd490d72c42000ca05357d0f926fa4a980d" Jul 17 18:48:32.680773 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:32.680559 2572 scope.go:117] "RemoveContainer" containerID="ab943ffa97dfe9a0a17ea7c16c305ff1427c10d813f23265b538fc70d6e88c6c" Jul 17 18:48:32.680773 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:48:32.680762 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6_odh-ai-gateway-infra(e17beeeb-6770-4dda-aaf6-4789918798db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6" podUID="e17beeeb-6770-4dda-aaf6-4789918798db" Jul 17 18:48:33.687282 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:33.687259 2572 scope.go:117] "RemoveContainer" containerID="ab943ffa97dfe9a0a17ea7c16c305ff1427c10d813f23265b538fc70d6e88c6c" Jul 17 18:48:33.687671 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:48:33.687451 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6_odh-ai-gateway-infra(e17beeeb-6770-4dda-aaf6-4789918798db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6" podUID="e17beeeb-6770-4dda-aaf6-4789918798db" Jul 17 18:48:37.742241 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:37.742202 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl"] Jul 17 18:48:37.746846 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:37.746826 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl" Jul 17 18:48:37.752444 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:37.752419 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl"] Jul 17 18:48:37.819644 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:37.819610 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ae2834-34b8-4ed8-a932-f216c1d4a89b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl\" (UID: \"22ae2834-34b8-4ed8-a932-f216c1d4a89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl" Jul 17 18:48:37.920567 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:37.920537 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ae2834-34b8-4ed8-a932-f216c1d4a89b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl\" (UID: \"22ae2834-34b8-4ed8-a932-f216c1d4a89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl" Jul 17 18:48:37.921160 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:37.921141 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ae2834-34b8-4ed8-a932-f216c1d4a89b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl\" (UID: \"22ae2834-34b8-4ed8-a932-f216c1d4a89b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl" Jul 17 18:48:38.058192 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:38.058159 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl" Jul 17 18:48:38.180363 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:38.180337 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl"] Jul 17 18:48:38.182553 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:48:38.182523 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ae2834_34b8_4ed8_a932_f216c1d4a89b.slice/crio-894cf536f7cf6e9aa27e3c58eadbe301aac10193e27794cc71b6d9ce1b6448d9 WatchSource:0}: Error finding container 894cf536f7cf6e9aa27e3c58eadbe301aac10193e27794cc71b6d9ce1b6448d9: Status 404 returned error can't find the container with id 894cf536f7cf6e9aa27e3c58eadbe301aac10193e27794cc71b6d9ce1b6448d9 Jul 17 18:48:38.712062 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:38.711961 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl" event={"ID":"22ae2834-34b8-4ed8-a932-f216c1d4a89b","Type":"ContainerStarted","Data":"0913baccf92ce6ef71ac86e514ba9e87ec23d46b79be4c3a04c0a5a6cf7dab48"} Jul 17 18:48:38.712062 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:38.712037 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl" event={"ID":"22ae2834-34b8-4ed8-a932-f216c1d4a89b","Type":"ContainerStarted","Data":"894cf536f7cf6e9aa27e3c58eadbe301aac10193e27794cc71b6d9ce1b6448d9"} Jul 17 18:48:38.729869 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:38.729825 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl" podStartSLOduration=1.729806774 podStartE2EDuration="1.729806774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:48:38.728857392 +0000 UTC m=+3114.138284999" watchObservedRunningTime="2026-07-17 18:48:38.729806774 +0000 UTC m=+3114.139234382" Jul 17 18:48:39.718816 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:39.718781 2572 generic.go:356] "Generic (PLEG): container finished" podID="22ae2834-34b8-4ed8-a932-f216c1d4a89b" containerID="0913baccf92ce6ef71ac86e514ba9e87ec23d46b79be4c3a04c0a5a6cf7dab48" exitCode=6 Jul 17 18:48:39.719320 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:39.718843 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl" event={"ID":"22ae2834-34b8-4ed8-a932-f216c1d4a89b","Type":"ContainerDied","Data":"0913baccf92ce6ef71ac86e514ba9e87ec23d46b79be4c3a04c0a5a6cf7dab48"} Jul 17 18:48:39.719320 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:39.719142 2572 scope.go:117] "RemoveContainer" containerID="0913baccf92ce6ef71ac86e514ba9e87ec23d46b79be4c3a04c0a5a6cf7dab48" Jul 17 18:48:40.725648 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:40.725551 2572 generic.go:356] "Generic (PLEG): container finished" podID="22ae2834-34b8-4ed8-a932-f216c1d4a89b" containerID="5e2e378ca1fef16cc655617c0a505ecab37283d271080b922c987392ca61ede8" exitCode=6 Jul 17 18:48:40.725648 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:40.725632 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl" event={"ID":"22ae2834-34b8-4ed8-a932-f216c1d4a89b","Type":"ContainerDied","Data":"5e2e378ca1fef16cc655617c0a505ecab37283d271080b922c987392ca61ede8"} Jul 17 18:48:40.726136 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:40.725720 2572 scope.go:117] "RemoveContainer" containerID="0913baccf92ce6ef71ac86e514ba9e87ec23d46b79be4c3a04c0a5a6cf7dab48" Jul 17 18:48:40.726136 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:40.725976 2572 scope.go:117] "RemoveContainer" containerID="5e2e378ca1fef16cc655617c0a505ecab37283d271080b922c987392ca61ede8" Jul 17 18:48:40.726251 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:48:40.726228 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl_odh-ai-gateway-infra(22ae2834-34b8-4ed8-a932-f216c1d4a89b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl" podUID="22ae2834-34b8-4ed8-a932-f216c1d4a89b" Jul 17 18:48:41.731718 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:41.731689 2572 scope.go:117] "RemoveContainer" containerID="5e2e378ca1fef16cc655617c0a505ecab37283d271080b922c987392ca61ede8" Jul 17 18:48:41.732130 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:48:41.731882 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl_odh-ai-gateway-infra(22ae2834-34b8-4ed8-a932-f216c1d4a89b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl" podUID="22ae2834-34b8-4ed8-a932-f216c1d4a89b" Jul 17 18:48:42.219447 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:42.219411 2572 scope.go:117] "RemoveContainer" containerID="f8874827ce7355f88fce086a9fc93eb03b05f16b22fa9bda2300115f4c6e5f79" Jul 17 18:48:42.737919 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:42.737885 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48" event={"ID":"ebb10b49-3cfc-4e29-9dd6-811b4ce85429","Type":"ContainerStarted","Data":"ed9ff4e415c4989fc51d8edbcfdf1abcd9ca73d2d007bd500e84577ee6eb9236"} Jul 17 18:48:42.753962 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:42.753910 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48" podStartSLOduration=17.753896195 podStartE2EDuration="17.753896195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:48:42.752379078 +0000 UTC m=+3118.161806697" watchObservedRunningTime="2026-07-17 18:48:42.753896195 +0000 UTC m=+3118.163323834" Jul 17 18:48:43.743932 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:43.743902 2572 generic.go:356] "Generic (PLEG): container finished" podID="ebb10b49-3cfc-4e29-9dd6-811b4ce85429" containerID="ed9ff4e415c4989fc51d8edbcfdf1abcd9ca73d2d007bd500e84577ee6eb9236" exitCode=6 Jul 17 18:48:43.744374 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:43.743979 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48" event={"ID":"ebb10b49-3cfc-4e29-9dd6-811b4ce85429","Type":"ContainerDied","Data":"ed9ff4e415c4989fc51d8edbcfdf1abcd9ca73d2d007bd500e84577ee6eb9236"} Jul 17 18:48:43.744374 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:43.744058 2572 scope.go:117] "RemoveContainer" containerID="f8874827ce7355f88fce086a9fc93eb03b05f16b22fa9bda2300115f4c6e5f79" Jul 17 18:48:43.744374 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:43.744290 2572 scope.go:117] "RemoveContainer" containerID="ed9ff4e415c4989fc51d8edbcfdf1abcd9ca73d2d007bd500e84577ee6eb9236" Jul 17 18:48:43.744542 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:48:43.744524 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48_odh-ai-gateway-infra(ebb10b49-3cfc-4e29-9dd6-811b4ce85429)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48" podUID="ebb10b49-3cfc-4e29-9dd6-811b4ce85429" Jul 17 18:48:43.765819 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:43.765797 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48"] Jul 17 18:48:44.890165 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:44.890142 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48" Jul 17 18:48:44.977070 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:44.977038 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb10b49-3cfc-4e29-9dd6-811b4ce85429-maas-api-service-ca\") pod \"ebb10b49-3cfc-4e29-9dd6-811b4ce85429\" (UID: \"ebb10b49-3cfc-4e29-9dd6-811b4ce85429\") " Jul 17 18:48:44.977380 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:44.977357 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb10b49-3cfc-4e29-9dd6-811b4ce85429-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ebb10b49-3cfc-4e29-9dd6-811b4ce85429" (UID: "ebb10b49-3cfc-4e29-9dd6-811b4ce85429"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:48:45.078568 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:45.078542 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb10b49-3cfc-4e29-9dd6-811b4ce85429-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:48:45.755579 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:45.755549 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48" Jul 17 18:48:45.755736 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:45.755576 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48" event={"ID":"ebb10b49-3cfc-4e29-9dd6-811b4ce85429","Type":"ContainerDied","Data":"4d8372101ac692ff7588bca11aca5ede2f250ad4147e6566d2de5ea705f46ca6"} Jul 17 18:48:45.755736 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:45.755640 2572 scope.go:117] "RemoveContainer" containerID="ed9ff4e415c4989fc51d8edbcfdf1abcd9ca73d2d007bd500e84577ee6eb9236" Jul 17 18:48:45.773048 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:45.773002 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48"] Jul 17 18:48:45.778368 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:45.778344 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-jrg48"] Jul 17 18:48:47.220062 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:47.220033 2572 scope.go:117] "RemoveContainer" containerID="ab943ffa97dfe9a0a17ea7c16c305ff1427c10d813f23265b538fc70d6e88c6c" Jul 17 18:48:47.224957 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:47.224932 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebb10b49-3cfc-4e29-9dd6-811b4ce85429" path="/var/lib/kubelet/pods/ebb10b49-3cfc-4e29-9dd6-811b4ce85429/volumes" Jul 17 18:48:47.767388 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:47.767348 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6" event={"ID":"e17beeeb-6770-4dda-aaf6-4789918798db","Type":"ContainerStarted","Data":"8de700248222647a75a4eb9975bfad75be31b0394d80b2fcf82a4bbc2a4bfa89"} Jul 17 18:48:47.786224 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:47.786178 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6" podStartSLOduration=18.786166075 podStartE2EDuration="18.786166075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:48:47.78585538 +0000 UTC m=+3123.195282987" watchObservedRunningTime="2026-07-17 18:48:47.786166075 +0000 UTC m=+3123.195593680" Jul 17 18:48:48.773569 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:48.773540 2572 generic.go:356] "Generic (PLEG): container finished" podID="e17beeeb-6770-4dda-aaf6-4789918798db" containerID="8de700248222647a75a4eb9975bfad75be31b0394d80b2fcf82a4bbc2a4bfa89" exitCode=7 Jul 17 18:48:48.773993 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:48.773614 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6" event={"ID":"e17beeeb-6770-4dda-aaf6-4789918798db","Type":"ContainerDied","Data":"8de700248222647a75a4eb9975bfad75be31b0394d80b2fcf82a4bbc2a4bfa89"} Jul 17 18:48:48.773993 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:48.773675 2572 scope.go:117] "RemoveContainer" containerID="ab943ffa97dfe9a0a17ea7c16c305ff1427c10d813f23265b538fc70d6e88c6c" Jul 17 18:48:48.773993 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:48.773961 2572 scope.go:117] "RemoveContainer" containerID="8de700248222647a75a4eb9975bfad75be31b0394d80b2fcf82a4bbc2a4bfa89" Jul 17 18:48:48.774240 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:48:48.774222 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6_odh-ai-gateway-infra(e17beeeb-6770-4dda-aaf6-4789918798db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6" podUID="e17beeeb-6770-4dda-aaf6-4789918798db" Jul 17 18:48:48.807895 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:48.807866 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6"] Jul 17 18:48:49.909826 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:49.909801 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6" Jul 17 18:48:50.022278 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:50.022248 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17beeeb-6770-4dda-aaf6-4789918798db-maas-api-service-ca\") pod \"e17beeeb-6770-4dda-aaf6-4789918798db\" (UID: \"e17beeeb-6770-4dda-aaf6-4789918798db\") " Jul 17 18:48:50.022596 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:50.022570 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e17beeeb-6770-4dda-aaf6-4789918798db-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e17beeeb-6770-4dda-aaf6-4789918798db" (UID: "e17beeeb-6770-4dda-aaf6-4789918798db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:48:50.123119 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:50.123055 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17beeeb-6770-4dda-aaf6-4789918798db-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:48:50.785067 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:50.785034 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6" Jul 17 18:48:50.785285 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:50.785063 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6" event={"ID":"e17beeeb-6770-4dda-aaf6-4789918798db","Type":"ContainerDied","Data":"a871ef56aabc5a222395744794a80684accd408cff7f589cfa4dcda78284c070"} Jul 17 18:48:50.785285 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:50.785136 2572 scope.go:117] "RemoveContainer" containerID="8de700248222647a75a4eb9975bfad75be31b0394d80b2fcf82a4bbc2a4bfa89" Jul 17 18:48:50.807261 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:50.807234 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6"] Jul 17 18:48:50.812100 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:50.812072 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-xshk6"] Jul 17 18:48:51.224971 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.224868 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e17beeeb-6770-4dda-aaf6-4789918798db" path="/var/lib/kubelet/pods/e17beeeb-6770-4dda-aaf6-4789918798db/volumes" Jul 17 18:48:51.233017 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.232989 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf"] Jul 17 18:48:51.233302 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.233288 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e17beeeb-6770-4dda-aaf6-4789918798db" containerName="revoke-keys" Jul 17 18:48:51.233356 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.233304 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e17beeeb-6770-4dda-aaf6-4789918798db" containerName="revoke-keys" Jul 17 18:48:51.233356 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.233314 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb10b49-3cfc-4e29-9dd6-811b4ce85429" containerName="revoke-keys" Jul 17 18:48:51.233356 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.233319 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb10b49-3cfc-4e29-9dd6-811b4ce85429" containerName="revoke-keys" Jul 17 18:48:51.233356 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.233337 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb10b49-3cfc-4e29-9dd6-811b4ce85429" containerName="revoke-keys" Jul 17 18:48:51.233356 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.233342 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb10b49-3cfc-4e29-9dd6-811b4ce85429" containerName="revoke-keys" Jul 17 18:48:51.233356 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.233349 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e17beeeb-6770-4dda-aaf6-4789918798db" containerName="revoke-keys" Jul 17 18:48:51.233356 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.233354 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e17beeeb-6770-4dda-aaf6-4789918798db" containerName="revoke-keys" Jul 17 18:48:51.233569 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.233399 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e17beeeb-6770-4dda-aaf6-4789918798db" containerName="revoke-keys" Jul 17 18:48:51.233569 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.233409 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb10b49-3cfc-4e29-9dd6-811b4ce85429" containerName="revoke-keys" Jul 17 18:48:51.233569 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.233414 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb10b49-3cfc-4e29-9dd6-811b4ce85429" containerName="revoke-keys" Jul 17 18:48:51.233569 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.233420 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb10b49-3cfc-4e29-9dd6-811b4ce85429" containerName="revoke-keys" Jul 17 18:48:51.233569 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.233427 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e17beeeb-6770-4dda-aaf6-4789918798db" containerName="revoke-keys" Jul 17 18:48:51.237796 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.237779 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf" Jul 17 18:48:51.245063 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.245040 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf"] Jul 17 18:48:51.335200 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.335162 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c3584b-7e25-4c15-b55b-1ce3967a4e75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf\" (UID: \"e8c3584b-7e25-4c15-b55b-1ce3967a4e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf" Jul 17 18:48:51.436266 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.436233 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c3584b-7e25-4c15-b55b-1ce3967a4e75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf\" (UID: \"e8c3584b-7e25-4c15-b55b-1ce3967a4e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf" Jul 17 18:48:51.436839 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.436817 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c3584b-7e25-4c15-b55b-1ce3967a4e75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf\" (UID: \"e8c3584b-7e25-4c15-b55b-1ce3967a4e75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf" Jul 17 18:48:51.548596 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.548563 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf" Jul 17 18:48:51.674273 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.674249 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf"] Jul 17 18:48:51.676079 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:48:51.676055 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c3584b_7e25_4c15_b55b_1ce3967a4e75.slice/crio-45ada47e654bc4c8c5956a003a2d7fc238fdcbcff76b59ce67a02a5bcadac7fb WatchSource:0}: Error finding container 45ada47e654bc4c8c5956a003a2d7fc238fdcbcff76b59ce67a02a5bcadac7fb: Status 404 returned error can't find the container with id 45ada47e654bc4c8c5956a003a2d7fc238fdcbcff76b59ce67a02a5bcadac7fb Jul 17 18:48:51.791664 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:51.791628 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf" event={"ID":"e8c3584b-7e25-4c15-b55b-1ce3967a4e75","Type":"ContainerStarted","Data":"45ada47e654bc4c8c5956a003a2d7fc238fdcbcff76b59ce67a02a5bcadac7fb"} Jul 17 18:48:52.798106 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:52.798070 2572 generic.go:356] "Generic (PLEG): container finished" podID="e8c3584b-7e25-4c15-b55b-1ce3967a4e75" containerID="785bb73e51419c5b0824d7e5d9e9f2be1f8725a1802fd9c4f8d99e65eaf90f88" exitCode=6 Jul 17 18:48:52.798469 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:52.798158 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf" event={"ID":"e8c3584b-7e25-4c15-b55b-1ce3967a4e75","Type":"ContainerDied","Data":"785bb73e51419c5b0824d7e5d9e9f2be1f8725a1802fd9c4f8d99e65eaf90f88"} Jul 17 18:48:52.798469 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:52.798313 2572 scope.go:117] "RemoveContainer" containerID="785bb73e51419c5b0824d7e5d9e9f2be1f8725a1802fd9c4f8d99e65eaf90f88" Jul 17 18:48:53.781300 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:53.781269 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2"] Jul 17 18:48:53.781590 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:53.781577 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb10b49-3cfc-4e29-9dd6-811b4ce85429" containerName="revoke-keys" Jul 17 18:48:53.781657 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:53.781591 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb10b49-3cfc-4e29-9dd6-811b4ce85429" containerName="revoke-keys" Jul 17 18:48:53.781657 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:53.781607 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e17beeeb-6770-4dda-aaf6-4789918798db" containerName="revoke-keys" Jul 17 18:48:53.781657 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:53.781613 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e17beeeb-6770-4dda-aaf6-4789918798db" containerName="revoke-keys" Jul 17 18:48:53.781763 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:53.781661 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e17beeeb-6770-4dda-aaf6-4789918798db" containerName="revoke-keys" Jul 17 18:48:53.784540 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:53.784525 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2" Jul 17 18:48:53.793168 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:53.793145 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2"] Jul 17 18:48:53.805501 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:53.805466 2572 generic.go:356] "Generic (PLEG): container finished" podID="e8c3584b-7e25-4c15-b55b-1ce3967a4e75" containerID="da1bc766fbea433debcb06b6cf2fbd583fb8a52e51e4963f936eff925be751bf" exitCode=6 Jul 17 18:48:53.805836 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:53.805556 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf" event={"ID":"e8c3584b-7e25-4c15-b55b-1ce3967a4e75","Type":"ContainerDied","Data":"da1bc766fbea433debcb06b6cf2fbd583fb8a52e51e4963f936eff925be751bf"} Jul 17 18:48:53.805836 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:53.805610 2572 scope.go:117] "RemoveContainer" containerID="785bb73e51419c5b0824d7e5d9e9f2be1f8725a1802fd9c4f8d99e65eaf90f88" Jul 17 18:48:53.805836 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:53.805819 2572 scope.go:117] "RemoveContainer" containerID="da1bc766fbea433debcb06b6cf2fbd583fb8a52e51e4963f936eff925be751bf" Jul 17 18:48:53.806092 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:48:53.806072 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf_odh-ai-gateway-infra(e8c3584b-7e25-4c15-b55b-1ce3967a4e75)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf" podUID="e8c3584b-7e25-4c15-b55b-1ce3967a4e75" Jul 17 18:48:53.956963 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:53.956926 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fac9284-7881-403d-83eb-aa68329b3126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-crkj2\" (UID: \"6fac9284-7881-403d-83eb-aa68329b3126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2" Jul 17 18:48:54.057564 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:54.057464 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fac9284-7881-403d-83eb-aa68329b3126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-crkj2\" (UID: \"6fac9284-7881-403d-83eb-aa68329b3126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2" Jul 17 18:48:54.058102 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:54.058084 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fac9284-7881-403d-83eb-aa68329b3126-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-crkj2\" (UID: \"6fac9284-7881-403d-83eb-aa68329b3126\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2" Jul 17 18:48:54.095620 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:54.095599 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2" Jul 17 18:48:54.222651 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:54.222625 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2"] Jul 17 18:48:54.224550 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:48:54.224523 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fac9284_7881_403d_83eb_aa68329b3126.slice/crio-8cf9b03b17aac929aaaf5a44fa6a023818e5d24d42a43f457a2294fa19f7c711 WatchSource:0}: Error finding container 8cf9b03b17aac929aaaf5a44fa6a023818e5d24d42a43f457a2294fa19f7c711: Status 404 returned error can't find the container with id 8cf9b03b17aac929aaaf5a44fa6a023818e5d24d42a43f457a2294fa19f7c711 Jul 17 18:48:54.811600 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:54.811570 2572 scope.go:117] "RemoveContainer" containerID="da1bc766fbea433debcb06b6cf2fbd583fb8a52e51e4963f936eff925be751bf" Jul 17 18:48:54.812066 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:48:54.811779 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf_odh-ai-gateway-infra(e8c3584b-7e25-4c15-b55b-1ce3967a4e75)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf" podUID="e8c3584b-7e25-4c15-b55b-1ce3967a4e75" Jul 17 18:48:54.812829 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:54.812807 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2" event={"ID":"6fac9284-7881-403d-83eb-aa68329b3126","Type":"ContainerStarted","Data":"3fd238b2e9261abd4e20b7e3f8875a536d34da272c9da42d5da44d826db46b09"} Jul 17 18:48:54.812879 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:54.812852 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2" event={"ID":"6fac9284-7881-403d-83eb-aa68329b3126","Type":"ContainerStarted","Data":"8cf9b03b17aac929aaaf5a44fa6a023818e5d24d42a43f457a2294fa19f7c711"} Jul 17 18:48:54.841231 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:54.841186 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2" podStartSLOduration=1.841172783 podStartE2EDuration="1.841172783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:48:54.839208651 +0000 UTC m=+3130.248636280" watchObservedRunningTime="2026-07-17 18:48:54.841172783 +0000 UTC m=+3130.250600389" Jul 17 18:48:55.818078 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:55.818045 2572 generic.go:356] "Generic (PLEG): container finished" podID="6fac9284-7881-403d-83eb-aa68329b3126" containerID="3fd238b2e9261abd4e20b7e3f8875a536d34da272c9da42d5da44d826db46b09" exitCode=6 Jul 17 18:48:55.818454 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:55.818096 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2" event={"ID":"6fac9284-7881-403d-83eb-aa68329b3126","Type":"ContainerDied","Data":"3fd238b2e9261abd4e20b7e3f8875a536d34da272c9da42d5da44d826db46b09"} Jul 17 18:48:55.818454 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:55.818308 2572 scope.go:117] "RemoveContainer" containerID="3fd238b2e9261abd4e20b7e3f8875a536d34da272c9da42d5da44d826db46b09" Jul 17 18:48:56.824045 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:56.823992 2572 generic.go:356] "Generic (PLEG): container finished" podID="6fac9284-7881-403d-83eb-aa68329b3126" containerID="3b8cc55957a266a97f91f60e67b0ce1731569bb04b406f304a7f21ca2fbe17b5" exitCode=6 Jul 17 18:48:56.824513 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:56.824081 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2" event={"ID":"6fac9284-7881-403d-83eb-aa68329b3126","Type":"ContainerDied","Data":"3b8cc55957a266a97f91f60e67b0ce1731569bb04b406f304a7f21ca2fbe17b5"} Jul 17 18:48:56.824513 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:56.824139 2572 scope.go:117] "RemoveContainer" containerID="3fd238b2e9261abd4e20b7e3f8875a536d34da272c9da42d5da44d826db46b09" Jul 17 18:48:56.824513 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:56.824293 2572 scope.go:117] "RemoveContainer" containerID="3b8cc55957a266a97f91f60e67b0ce1731569bb04b406f304a7f21ca2fbe17b5" Jul 17 18:48:56.824513 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:48:56.824501 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-crkj2_odh-ai-gateway-infra(6fac9284-7881-403d-83eb-aa68329b3126)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2" podUID="6fac9284-7881-403d-83eb-aa68329b3126" Jul 17 18:48:57.219982 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:57.219906 2572 scope.go:117] "RemoveContainer" containerID="5e2e378ca1fef16cc655617c0a505ecab37283d271080b922c987392ca61ede8" Jul 17 18:48:57.830669 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:57.830633 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl" event={"ID":"22ae2834-34b8-4ed8-a932-f216c1d4a89b","Type":"ContainerStarted","Data":"b2572c9004c94e68a5d7a451f16ca06918c32060582f7bd5bd64fe1611dd1ad3"} Jul 17 18:48:57.833793 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:57.832821 2572 scope.go:117] "RemoveContainer" containerID="3b8cc55957a266a97f91f60e67b0ce1731569bb04b406f304a7f21ca2fbe17b5" Jul 17 18:48:57.833793 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:48:57.833160 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-crkj2_odh-ai-gateway-infra(6fac9284-7881-403d-83eb-aa68329b3126)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2" podUID="6fac9284-7881-403d-83eb-aa68329b3126" Jul 17 18:48:58.838437 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:58.838403 2572 generic.go:356] "Generic (PLEG): container finished" podID="22ae2834-34b8-4ed8-a932-f216c1d4a89b" containerID="b2572c9004c94e68a5d7a451f16ca06918c32060582f7bd5bd64fe1611dd1ad3" exitCode=6 Jul 17 18:48:58.838893 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:58.838478 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl" event={"ID":"22ae2834-34b8-4ed8-a932-f216c1d4a89b","Type":"ContainerDied","Data":"b2572c9004c94e68a5d7a451f16ca06918c32060582f7bd5bd64fe1611dd1ad3"} Jul 17 18:48:58.838893 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:58.838563 2572 scope.go:117] "RemoveContainer" containerID="5e2e378ca1fef16cc655617c0a505ecab37283d271080b922c987392ca61ede8" Jul 17 18:48:58.838893 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:58.838836 2572 scope.go:117] "RemoveContainer" containerID="b2572c9004c94e68a5d7a451f16ca06918c32060582f7bd5bd64fe1611dd1ad3" Jul 17 18:48:58.839109 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:48:58.839090 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl_odh-ai-gateway-infra(22ae2834-34b8-4ed8-a932-f216c1d4a89b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl" podUID="22ae2834-34b8-4ed8-a932-f216c1d4a89b" Jul 17 18:48:58.856249 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:58.856219 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl"] Jul 17 18:48:59.980099 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:48:59.980075 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl" Jul 17 18:49:00.112362 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:00.112258 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ae2834-34b8-4ed8-a932-f216c1d4a89b-maas-api-service-ca\") pod \"22ae2834-34b8-4ed8-a932-f216c1d4a89b\" (UID: \"22ae2834-34b8-4ed8-a932-f216c1d4a89b\") " Jul 17 18:49:00.112654 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:00.112631 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22ae2834-34b8-4ed8-a932-f216c1d4a89b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22ae2834-34b8-4ed8-a932-f216c1d4a89b" (UID: "22ae2834-34b8-4ed8-a932-f216c1d4a89b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:49:00.213300 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:00.213266 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22ae2834-34b8-4ed8-a932-f216c1d4a89b-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:49:00.850385 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:00.850354 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl" Jul 17 18:49:00.850588 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:00.850397 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl" event={"ID":"22ae2834-34b8-4ed8-a932-f216c1d4a89b","Type":"ContainerDied","Data":"894cf536f7cf6e9aa27e3c58eadbe301aac10193e27794cc71b6d9ce1b6448d9"} Jul 17 18:49:00.850588 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:00.850465 2572 scope.go:117] "RemoveContainer" containerID="b2572c9004c94e68a5d7a451f16ca06918c32060582f7bd5bd64fe1611dd1ad3" Jul 17 18:49:00.873596 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:00.873574 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl"] Jul 17 18:49:00.879125 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:00.879103 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9wzl"] Jul 17 18:49:01.225038 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:01.224945 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22ae2834-34b8-4ed8-a932-f216c1d4a89b" path="/var/lib/kubelet/pods/22ae2834-34b8-4ed8-a932-f216c1d4a89b/volumes" Jul 17 18:49:06.219902 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:06.219829 2572 scope.go:117] "RemoveContainer" containerID="da1bc766fbea433debcb06b6cf2fbd583fb8a52e51e4963f936eff925be751bf" Jul 17 18:49:06.884969 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:06.884930 2572 generic.go:356] "Generic (PLEG): container finished" podID="e8c3584b-7e25-4c15-b55b-1ce3967a4e75" containerID="db349aa096d86bb2cc536bc6399387c3f911269aa9ca559cc40fc35227813e20" exitCode=6 Jul 17 18:49:06.885134 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:06.885003 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf" event={"ID":"e8c3584b-7e25-4c15-b55b-1ce3967a4e75","Type":"ContainerDied","Data":"db349aa096d86bb2cc536bc6399387c3f911269aa9ca559cc40fc35227813e20"} Jul 17 18:49:06.885134 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:06.885082 2572 scope.go:117] "RemoveContainer" containerID="da1bc766fbea433debcb06b6cf2fbd583fb8a52e51e4963f936eff925be751bf" Jul 17 18:49:06.885390 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:06.885374 2572 scope.go:117] "RemoveContainer" containerID="db349aa096d86bb2cc536bc6399387c3f911269aa9ca559cc40fc35227813e20" Jul 17 18:49:06.885632 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:49:06.885614 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf_odh-ai-gateway-infra(e8c3584b-7e25-4c15-b55b-1ce3967a4e75)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf" podUID="e8c3584b-7e25-4c15-b55b-1ce3967a4e75" Jul 17 18:49:07.909086 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:07.909053 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf"] Jul 17 18:49:08.040300 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:08.040276 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf" Jul 17 18:49:08.074602 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:08.074560 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c3584b-7e25-4c15-b55b-1ce3967a4e75-maas-api-service-ca\") pod \"e8c3584b-7e25-4c15-b55b-1ce3967a4e75\" (UID: \"e8c3584b-7e25-4c15-b55b-1ce3967a4e75\") " Jul 17 18:49:08.074925 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:08.074902 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c3584b-7e25-4c15-b55b-1ce3967a4e75-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e8c3584b-7e25-4c15-b55b-1ce3967a4e75" (UID: "e8c3584b-7e25-4c15-b55b-1ce3967a4e75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:49:08.175392 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:08.175296 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c3584b-7e25-4c15-b55b-1ce3967a4e75-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:49:08.895797 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:08.895769 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf" Jul 17 18:49:08.895797 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:08.895773 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf" event={"ID":"e8c3584b-7e25-4c15-b55b-1ce3967a4e75","Type":"ContainerDied","Data":"45ada47e654bc4c8c5956a003a2d7fc238fdcbcff76b59ce67a02a5bcadac7fb"} Jul 17 18:49:08.896092 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:08.895837 2572 scope.go:117] "RemoveContainer" containerID="db349aa096d86bb2cc536bc6399387c3f911269aa9ca559cc40fc35227813e20" Jul 17 18:49:08.916900 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:08.916872 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf"] Jul 17 18:49:08.921894 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:08.921868 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-kljwf"] Jul 17 18:49:09.224698 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:09.224619 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8c3584b-7e25-4c15-b55b-1ce3967a4e75" path="/var/lib/kubelet/pods/e8c3584b-7e25-4c15-b55b-1ce3967a4e75/volumes" Jul 17 18:49:13.220333 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:13.220299 2572 scope.go:117] "RemoveContainer" containerID="3b8cc55957a266a97f91f60e67b0ce1731569bb04b406f304a7f21ca2fbe17b5" Jul 17 18:49:13.919763 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:13.919731 2572 generic.go:356] "Generic (PLEG): container finished" podID="6fac9284-7881-403d-83eb-aa68329b3126" containerID="2f1dd03c9673a3da91cd4bf2784c207003109c5efe7be9824cc626013c02149c" exitCode=6 Jul 17 18:49:13.919943 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:13.919809 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2" event={"ID":"6fac9284-7881-403d-83eb-aa68329b3126","Type":"ContainerDied","Data":"2f1dd03c9673a3da91cd4bf2784c207003109c5efe7be9824cc626013c02149c"} Jul 17 18:49:13.919943 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:13.919870 2572 scope.go:117] "RemoveContainer" containerID="3b8cc55957a266a97f91f60e67b0ce1731569bb04b406f304a7f21ca2fbe17b5" Jul 17 18:49:13.920167 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:13.920151 2572 scope.go:117] "RemoveContainer" containerID="2f1dd03c9673a3da91cd4bf2784c207003109c5efe7be9824cc626013c02149c" Jul 17 18:49:13.920378 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:49:13.920361 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-crkj2_odh-ai-gateway-infra(6fac9284-7881-403d-83eb-aa68329b3126)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2" podUID="6fac9284-7881-403d-83eb-aa68329b3126" Jul 17 18:49:14.945351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:14.945313 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2"] Jul 17 18:49:15.073834 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:15.073811 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2" Jul 17 18:49:15.130099 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:15.130061 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fac9284-7881-403d-83eb-aa68329b3126-maas-api-service-ca\") pod \"6fac9284-7881-403d-83eb-aa68329b3126\" (UID: \"6fac9284-7881-403d-83eb-aa68329b3126\") " Jul 17 18:49:15.130410 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:15.130388 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fac9284-7881-403d-83eb-aa68329b3126-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6fac9284-7881-403d-83eb-aa68329b3126" (UID: "6fac9284-7881-403d-83eb-aa68329b3126"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:49:15.231203 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:15.231130 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fac9284-7881-403d-83eb-aa68329b3126-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:49:15.931108 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:15.931075 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2" event={"ID":"6fac9284-7881-403d-83eb-aa68329b3126","Type":"ContainerDied","Data":"8cf9b03b17aac929aaaf5a44fa6a023818e5d24d42a43f457a2294fa19f7c711"} Jul 17 18:49:15.931108 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:15.931108 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2" Jul 17 18:49:15.931342 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:15.931135 2572 scope.go:117] "RemoveContainer" containerID="2f1dd03c9673a3da91cd4bf2784c207003109c5efe7be9824cc626013c02149c" Jul 17 18:49:15.947713 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:15.947689 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2"] Jul 17 18:49:15.951219 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:15.951197 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-crkj2"] Jul 17 18:49:17.224137 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:17.224109 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fac9284-7881-403d-83eb-aa68329b3126" path="/var/lib/kubelet/pods/6fac9284-7881-403d-83eb-aa68329b3126/volumes" Jul 17 18:49:25.657835 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.657800 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7"] Jul 17 18:49:25.658323 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658273 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22ae2834-34b8-4ed8-a932-f216c1d4a89b" containerName="revoke-keys" Jul 17 18:49:25.658323 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658291 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="22ae2834-34b8-4ed8-a932-f216c1d4a89b" containerName="revoke-keys" Jul 17 18:49:25.658323 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658314 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c3584b-7e25-4c15-b55b-1ce3967a4e75" containerName="revoke-keys" Jul 17 18:49:25.658323 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658322 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c3584b-7e25-4c15-b55b-1ce3967a4e75" containerName="revoke-keys" Jul 17 18:49:25.658543 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658330 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c3584b-7e25-4c15-b55b-1ce3967a4e75" containerName="revoke-keys" Jul 17 18:49:25.658543 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658339 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c3584b-7e25-4c15-b55b-1ce3967a4e75" containerName="revoke-keys" Jul 17 18:49:25.658543 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658355 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22ae2834-34b8-4ed8-a932-f216c1d4a89b" containerName="revoke-keys" Jul 17 18:49:25.658543 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658363 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="22ae2834-34b8-4ed8-a932-f216c1d4a89b" containerName="revoke-keys" Jul 17 18:49:25.658543 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658375 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fac9284-7881-403d-83eb-aa68329b3126" containerName="revoke-keys" Jul 17 18:49:25.658543 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658385 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fac9284-7881-403d-83eb-aa68329b3126" containerName="revoke-keys" Jul 17 18:49:25.658543 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658399 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fac9284-7881-403d-83eb-aa68329b3126" containerName="revoke-keys" Jul 17 18:49:25.658543 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658407 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fac9284-7881-403d-83eb-aa68329b3126" containerName="revoke-keys" Jul 17 18:49:25.658543 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658416 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fac9284-7881-403d-83eb-aa68329b3126" containerName="revoke-keys" Jul 17 18:49:25.658543 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658423 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fac9284-7881-403d-83eb-aa68329b3126" containerName="revoke-keys" Jul 17 18:49:25.658543 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658498 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c3584b-7e25-4c15-b55b-1ce3967a4e75" containerName="revoke-keys" Jul 17 18:49:25.658543 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658509 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c3584b-7e25-4c15-b55b-1ce3967a4e75" containerName="revoke-keys" Jul 17 18:49:25.658543 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658519 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c3584b-7e25-4c15-b55b-1ce3967a4e75" containerName="revoke-keys" Jul 17 18:49:25.658543 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658530 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="22ae2834-34b8-4ed8-a932-f216c1d4a89b" containerName="revoke-keys" Jul 17 18:49:25.658543 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658538 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fac9284-7881-403d-83eb-aa68329b3126" containerName="revoke-keys" Jul 17 18:49:25.658543 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658549 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fac9284-7881-403d-83eb-aa68329b3126" containerName="revoke-keys" Jul 17 18:49:25.659313 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658558 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fac9284-7881-403d-83eb-aa68329b3126" containerName="revoke-keys" Jul 17 18:49:25.659313 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.658568 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="22ae2834-34b8-4ed8-a932-f216c1d4a89b" containerName="revoke-keys" Jul 17 18:49:25.661627 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.661606 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7" Jul 17 18:49:25.664218 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.664201 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:49:25.664303 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.664213 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:49:25.668087 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.668030 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7"] Jul 17 18:49:25.726495 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.726465 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa12e8f-8c4f-42ac-87c6-10ec039d824a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7\" (UID: \"cfa12e8f-8c4f-42ac-87c6-10ec039d824a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7" Jul 17 18:49:25.827169 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.827143 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa12e8f-8c4f-42ac-87c6-10ec039d824a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7\" (UID: \"cfa12e8f-8c4f-42ac-87c6-10ec039d824a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7" Jul 17 18:49:25.827686 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.827670 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa12e8f-8c4f-42ac-87c6-10ec039d824a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7\" (UID: \"cfa12e8f-8c4f-42ac-87c6-10ec039d824a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7" Jul 17 18:49:25.972593 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:25.972519 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7" Jul 17 18:49:26.097711 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:26.097685 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7"] Jul 17 18:49:26.099997 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:49:26.099967 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa12e8f_8c4f_42ac_87c6_10ec039d824a.slice/crio-bd816dc14e1f3c29e10a37c7d4f07f503eb67e2755459a9f316592c30799b1a8 WatchSource:0}: Error finding container bd816dc14e1f3c29e10a37c7d4f07f503eb67e2755459a9f316592c30799b1a8: Status 404 returned error can't find the container with id bd816dc14e1f3c29e10a37c7d4f07f503eb67e2755459a9f316592c30799b1a8 Jul 17 18:49:26.983239 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:26.983207 2572 generic.go:356] "Generic (PLEG): container finished" podID="cfa12e8f-8c4f-42ac-87c6-10ec039d824a" containerID="7b53828d0bda271969961ee6ed130e2d0ed90334ad5e7194fd99d1ad7a5d9c73" exitCode=6 Jul 17 18:49:26.983558 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:26.983295 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7" event={"ID":"cfa12e8f-8c4f-42ac-87c6-10ec039d824a","Type":"ContainerDied","Data":"7b53828d0bda271969961ee6ed130e2d0ed90334ad5e7194fd99d1ad7a5d9c73"} Jul 17 18:49:26.983558 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:26.983349 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7" event={"ID":"cfa12e8f-8c4f-42ac-87c6-10ec039d824a","Type":"ContainerStarted","Data":"bd816dc14e1f3c29e10a37c7d4f07f503eb67e2755459a9f316592c30799b1a8"} Jul 17 18:49:26.983558 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:26.983467 2572 scope.go:117] "RemoveContainer" containerID="7b53828d0bda271969961ee6ed130e2d0ed90334ad5e7194fd99d1ad7a5d9c73" Jul 17 18:49:27.990024 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:27.989927 2572 generic.go:356] "Generic (PLEG): container finished" podID="cfa12e8f-8c4f-42ac-87c6-10ec039d824a" containerID="e658b34e956e85181d9dbaf906805c9f541884a3428687a277a09fae38e742ba" exitCode=6 Jul 17 18:49:27.990447 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:27.990003 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7" event={"ID":"cfa12e8f-8c4f-42ac-87c6-10ec039d824a","Type":"ContainerDied","Data":"e658b34e956e85181d9dbaf906805c9f541884a3428687a277a09fae38e742ba"} Jul 17 18:49:27.990447 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:27.990079 2572 scope.go:117] "RemoveContainer" containerID="7b53828d0bda271969961ee6ed130e2d0ed90334ad5e7194fd99d1ad7a5d9c73" Jul 17 18:49:27.990447 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:27.990327 2572 scope.go:117] "RemoveContainer" containerID="e658b34e956e85181d9dbaf906805c9f541884a3428687a277a09fae38e742ba" Jul 17 18:49:27.990566 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:49:27.990546 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7_odh-ai-gateway-infra(cfa12e8f-8c4f-42ac-87c6-10ec039d824a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7" podUID="cfa12e8f-8c4f-42ac-87c6-10ec039d824a" Jul 17 18:49:28.996278 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:28.996252 2572 scope.go:117] "RemoveContainer" containerID="e658b34e956e85181d9dbaf906805c9f541884a3428687a277a09fae38e742ba" Jul 17 18:49:28.996676 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:49:28.996433 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7_odh-ai-gateway-infra(cfa12e8f-8c4f-42ac-87c6-10ec039d824a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7" podUID="cfa12e8f-8c4f-42ac-87c6-10ec039d824a" Jul 17 18:49:30.045838 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:30.045803 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm"] Jul 17 18:49:30.046254 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:30.046110 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22ae2834-34b8-4ed8-a932-f216c1d4a89b" containerName="revoke-keys" Jul 17 18:49:30.046254 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:30.046122 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="22ae2834-34b8-4ed8-a932-f216c1d4a89b" containerName="revoke-keys" Jul 17 18:49:30.046254 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:30.046137 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c3584b-7e25-4c15-b55b-1ce3967a4e75" containerName="revoke-keys" Jul 17 18:49:30.046254 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:30.046143 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c3584b-7e25-4c15-b55b-1ce3967a4e75" containerName="revoke-keys" Jul 17 18:49:30.046254 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:30.046211 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="22ae2834-34b8-4ed8-a932-f216c1d4a89b" containerName="revoke-keys" Jul 17 18:49:30.050395 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:30.050377 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm" Jul 17 18:49:30.056203 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:30.056172 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm"] Jul 17 18:49:30.162388 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:30.162355 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0822b07e-213b-4a16-a3d6-f2b51846c5d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm\" (UID: \"0822b07e-213b-4a16-a3d6-f2b51846c5d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm" Jul 17 18:49:30.263415 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:30.263381 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0822b07e-213b-4a16-a3d6-f2b51846c5d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm\" (UID: \"0822b07e-213b-4a16-a3d6-f2b51846c5d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm" Jul 17 18:49:30.263981 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:30.263961 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0822b07e-213b-4a16-a3d6-f2b51846c5d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm\" (UID: \"0822b07e-213b-4a16-a3d6-f2b51846c5d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm" Jul 17 18:49:30.363141 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:30.363051 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm" Jul 17 18:49:30.486440 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:30.486414 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm"] Jul 17 18:49:30.488810 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:49:30.488777 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0822b07e_213b_4a16_a3d6_f2b51846c5d5.slice/crio-50321b9df1890df0ac53578ba3fec3aac10eb7b5ee771f42f735082197ae8bed WatchSource:0}: Error finding container 50321b9df1890df0ac53578ba3fec3aac10eb7b5ee771f42f735082197ae8bed: Status 404 returned error can't find the container with id 50321b9df1890df0ac53578ba3fec3aac10eb7b5ee771f42f735082197ae8bed Jul 17 18:49:31.006189 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:31.006103 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm" event={"ID":"0822b07e-213b-4a16-a3d6-f2b51846c5d5","Type":"ContainerStarted","Data":"00ef97a54f0f278a994b5f20b2e60a7595db56848c5ba038631f85486dc30686"} Jul 17 18:49:31.006189 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:31.006161 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm" event={"ID":"0822b07e-213b-4a16-a3d6-f2b51846c5d5","Type":"ContainerStarted","Data":"50321b9df1890df0ac53578ba3fec3aac10eb7b5ee771f42f735082197ae8bed"} Jul 17 18:49:31.021283 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:31.021239 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm" podStartSLOduration=1.021224101 podStartE2EDuration="1.021224101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:49:31.020105167 +0000 UTC m=+3166.429532773" watchObservedRunningTime="2026-07-17 18:49:31.021224101 +0000 UTC m=+3166.430651707" Jul 17 18:49:32.012366 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:32.012333 2572 generic.go:356] "Generic (PLEG): container finished" podID="0822b07e-213b-4a16-a3d6-f2b51846c5d5" containerID="00ef97a54f0f278a994b5f20b2e60a7595db56848c5ba038631f85486dc30686" exitCode=7 Jul 17 18:49:32.012789 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:32.012414 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm" event={"ID":"0822b07e-213b-4a16-a3d6-f2b51846c5d5","Type":"ContainerDied","Data":"00ef97a54f0f278a994b5f20b2e60a7595db56848c5ba038631f85486dc30686"} Jul 17 18:49:32.012789 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:32.012539 2572 scope.go:117] "RemoveContainer" containerID="00ef97a54f0f278a994b5f20b2e60a7595db56848c5ba038631f85486dc30686" Jul 17 18:49:33.018866 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:33.018776 2572 generic.go:356] "Generic (PLEG): container finished" podID="0822b07e-213b-4a16-a3d6-f2b51846c5d5" containerID="93fa8b31bbed72f40e1ec4096a36d79fb9d20da83d410f91c172903f5688bae3" exitCode=7 Jul 17 18:49:33.019307 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:33.018872 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm" event={"ID":"0822b07e-213b-4a16-a3d6-f2b51846c5d5","Type":"ContainerDied","Data":"93fa8b31bbed72f40e1ec4096a36d79fb9d20da83d410f91c172903f5688bae3"} Jul 17 18:49:33.019307 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:33.018933 2572 scope.go:117] "RemoveContainer" containerID="00ef97a54f0f278a994b5f20b2e60a7595db56848c5ba038631f85486dc30686" Jul 17 18:49:33.019307 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:33.019166 2572 scope.go:117] "RemoveContainer" containerID="93fa8b31bbed72f40e1ec4096a36d79fb9d20da83d410f91c172903f5688bae3" Jul 17 18:49:33.019450 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:49:33.019342 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm_odh-ai-gateway-infra(0822b07e-213b-4a16-a3d6-f2b51846c5d5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm" podUID="0822b07e-213b-4a16-a3d6-f2b51846c5d5" Jul 17 18:49:34.025069 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:34.025042 2572 scope.go:117] "RemoveContainer" containerID="93fa8b31bbed72f40e1ec4096a36d79fb9d20da83d410f91c172903f5688bae3" Jul 17 18:49:34.025450 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:49:34.025224 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm_odh-ai-gateway-infra(0822b07e-213b-4a16-a3d6-f2b51846c5d5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm" podUID="0822b07e-213b-4a16-a3d6-f2b51846c5d5" Jul 17 18:49:37.787247 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:37.787208 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx"] Jul 17 18:49:37.791687 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:37.791671 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx" Jul 17 18:49:37.796641 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:37.796612 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx"] Jul 17 18:49:37.928593 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:37.928560 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60bf3694-dcfc-4035-8373-8877f3dff5ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx\" (UID: \"60bf3694-dcfc-4035-8373-8877f3dff5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx" Jul 17 18:49:38.029223 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:38.029189 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60bf3694-dcfc-4035-8373-8877f3dff5ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx\" (UID: \"60bf3694-dcfc-4035-8373-8877f3dff5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx" Jul 17 18:49:38.029864 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:38.029842 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60bf3694-dcfc-4035-8373-8877f3dff5ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx\" (UID: \"60bf3694-dcfc-4035-8373-8877f3dff5ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx" Jul 17 18:49:38.103066 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:38.102980 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx" Jul 17 18:49:38.229854 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:38.229800 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx"] Jul 17 18:49:38.231929 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:49:38.231907 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60bf3694_dcfc_4035_8373_8877f3dff5ea.slice/crio-f25725d24da433313e3954184be8431243abb09455bef3bee8646c4186fbb218 WatchSource:0}: Error finding container f25725d24da433313e3954184be8431243abb09455bef3bee8646c4186fbb218: Status 404 returned error can't find the container with id f25725d24da433313e3954184be8431243abb09455bef3bee8646c4186fbb218 Jul 17 18:49:39.049786 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:39.049753 2572 generic.go:356] "Generic (PLEG): container finished" podID="60bf3694-dcfc-4035-8373-8877f3dff5ea" containerID="ff6818af71f0b156d9b2c80c7f2bf1e18e72c92c4ba101b7ed5a72648a21bed6" exitCode=6 Jul 17 18:49:39.050172 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:39.049842 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx" event={"ID":"60bf3694-dcfc-4035-8373-8877f3dff5ea","Type":"ContainerDied","Data":"ff6818af71f0b156d9b2c80c7f2bf1e18e72c92c4ba101b7ed5a72648a21bed6"} Jul 17 18:49:39.050172 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:39.049903 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx" event={"ID":"60bf3694-dcfc-4035-8373-8877f3dff5ea","Type":"ContainerStarted","Data":"f25725d24da433313e3954184be8431243abb09455bef3bee8646c4186fbb218"} Jul 17 18:49:39.050172 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:39.050026 2572 scope.go:117] "RemoveContainer" containerID="ff6818af71f0b156d9b2c80c7f2bf1e18e72c92c4ba101b7ed5a72648a21bed6" Jul 17 18:49:40.056144 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:40.056113 2572 generic.go:356] "Generic (PLEG): container finished" podID="60bf3694-dcfc-4035-8373-8877f3dff5ea" containerID="c50042deefe600803ff83bc6c4d55a1831c3b461887390a900c8b083511f3e15" exitCode=6 Jul 17 18:49:40.056550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:40.056241 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx" event={"ID":"60bf3694-dcfc-4035-8373-8877f3dff5ea","Type":"ContainerDied","Data":"c50042deefe600803ff83bc6c4d55a1831c3b461887390a900c8b083511f3e15"} Jul 17 18:49:40.056550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:40.056308 2572 scope.go:117] "RemoveContainer" containerID="ff6818af71f0b156d9b2c80c7f2bf1e18e72c92c4ba101b7ed5a72648a21bed6" Jul 17 18:49:40.056550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:40.056390 2572 scope.go:117] "RemoveContainer" containerID="c50042deefe600803ff83bc6c4d55a1831c3b461887390a900c8b083511f3e15" Jul 17 18:49:40.056720 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:49:40.056583 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx_odh-ai-gateway-infra(60bf3694-dcfc-4035-8373-8877f3dff5ea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx" podUID="60bf3694-dcfc-4035-8373-8877f3dff5ea" Jul 17 18:49:41.062576 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:41.062549 2572 scope.go:117] "RemoveContainer" containerID="c50042deefe600803ff83bc6c4d55a1831c3b461887390a900c8b083511f3e15" Jul 17 18:49:41.062984 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:49:41.062732 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx_odh-ai-gateway-infra(60bf3694-dcfc-4035-8373-8877f3dff5ea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx" podUID="60bf3694-dcfc-4035-8373-8877f3dff5ea" Jul 17 18:49:41.272427 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:41.272393 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx"] Jul 17 18:49:41.276671 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:41.276655 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx" Jul 17 18:49:41.283608 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:41.283370 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx"] Jul 17 18:49:41.358230 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:41.358148 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f699b76-b26f-4579-b271-3095a8ec4194-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx\" (UID: \"2f699b76-b26f-4579-b271-3095a8ec4194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx" Jul 17 18:49:41.459316 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:41.459283 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f699b76-b26f-4579-b271-3095a8ec4194-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx\" (UID: \"2f699b76-b26f-4579-b271-3095a8ec4194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx" Jul 17 18:49:41.459817 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:41.459800 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f699b76-b26f-4579-b271-3095a8ec4194-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx\" (UID: \"2f699b76-b26f-4579-b271-3095a8ec4194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx" Jul 17 18:49:41.588176 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:41.588146 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx" Jul 17 18:49:41.711187 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:41.711134 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx"] Jul 17 18:49:41.713683 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:49:41.713654 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f699b76_b26f_4579_b271_3095a8ec4194.slice/crio-f2906624d9678cc6893b40c87c0d53c3fb7e1c242f0f4aa8b3459740b4dbfe43 WatchSource:0}: Error finding container f2906624d9678cc6893b40c87c0d53c3fb7e1c242f0f4aa8b3459740b4dbfe43: Status 404 returned error can't find the container with id f2906624d9678cc6893b40c87c0d53c3fb7e1c242f0f4aa8b3459740b4dbfe43 Jul 17 18:49:42.068378 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:42.068347 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx" event={"ID":"2f699b76-b26f-4579-b271-3095a8ec4194","Type":"ContainerStarted","Data":"3e56fef9702945cba9edb06a86da75d27e86d5bd5802b67f86fa87fbd26313a0"} Jul 17 18:49:42.068706 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:42.068402 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx" event={"ID":"2f699b76-b26f-4579-b271-3095a8ec4194","Type":"ContainerStarted","Data":"f2906624d9678cc6893b40c87c0d53c3fb7e1c242f0f4aa8b3459740b4dbfe43"} Jul 17 18:49:42.082725 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:42.082681 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx" podStartSLOduration=1.0826631 podStartE2EDuration="1.0826631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:49:42.082227014 +0000 UTC m=+3177.491654639" watchObservedRunningTime="2026-07-17 18:49:42.0826631 +0000 UTC m=+3177.492090705" Jul 17 18:49:42.220344 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:42.220272 2572 scope.go:117] "RemoveContainer" containerID="e658b34e956e85181d9dbaf906805c9f541884a3428687a277a09fae38e742ba" Jul 17 18:49:43.075063 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:43.075030 2572 generic.go:356] "Generic (PLEG): container finished" podID="2f699b76-b26f-4579-b271-3095a8ec4194" containerID="3e56fef9702945cba9edb06a86da75d27e86d5bd5802b67f86fa87fbd26313a0" exitCode=6 Jul 17 18:49:43.075468 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:43.075118 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx" event={"ID":"2f699b76-b26f-4579-b271-3095a8ec4194","Type":"ContainerDied","Data":"3e56fef9702945cba9edb06a86da75d27e86d5bd5802b67f86fa87fbd26313a0"} Jul 17 18:49:43.075468 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:43.075271 2572 scope.go:117] "RemoveContainer" containerID="3e56fef9702945cba9edb06a86da75d27e86d5bd5802b67f86fa87fbd26313a0" Jul 17 18:49:43.077060 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:43.077038 2572 generic.go:356] "Generic (PLEG): container finished" podID="cfa12e8f-8c4f-42ac-87c6-10ec039d824a" containerID="1093f40f30cc5adeda421ffb88d3551c0283df8c367ad63bcb551b33620ebf81" exitCode=6 Jul 17 18:49:43.077163 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:43.077067 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7" event={"ID":"cfa12e8f-8c4f-42ac-87c6-10ec039d824a","Type":"ContainerDied","Data":"1093f40f30cc5adeda421ffb88d3551c0283df8c367ad63bcb551b33620ebf81"} Jul 17 18:49:43.077163 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:43.077105 2572 scope.go:117] "RemoveContainer" containerID="e658b34e956e85181d9dbaf906805c9f541884a3428687a277a09fae38e742ba" Jul 17 18:49:43.077422 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:43.077402 2572 scope.go:117] "RemoveContainer" containerID="1093f40f30cc5adeda421ffb88d3551c0283df8c367ad63bcb551b33620ebf81" Jul 17 18:49:43.077639 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:49:43.077616 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7_odh-ai-gateway-infra(cfa12e8f-8c4f-42ac-87c6-10ec039d824a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7" podUID="cfa12e8f-8c4f-42ac-87c6-10ec039d824a" Jul 17 18:49:44.085904 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:44.085869 2572 generic.go:356] "Generic (PLEG): container finished" podID="2f699b76-b26f-4579-b271-3095a8ec4194" containerID="2bf7c67e71ff52553fff7054cd8984c3bec360e90e0b72ed9bc48fc7189bdbb7" exitCode=6 Jul 17 18:49:44.086406 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:44.085927 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx" event={"ID":"2f699b76-b26f-4579-b271-3095a8ec4194","Type":"ContainerDied","Data":"2bf7c67e71ff52553fff7054cd8984c3bec360e90e0b72ed9bc48fc7189bdbb7"} Jul 17 18:49:44.086406 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:44.085975 2572 scope.go:117] "RemoveContainer" containerID="3e56fef9702945cba9edb06a86da75d27e86d5bd5802b67f86fa87fbd26313a0" Jul 17 18:49:44.086406 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:44.086195 2572 scope.go:117] "RemoveContainer" containerID="2bf7c67e71ff52553fff7054cd8984c3bec360e90e0b72ed9bc48fc7189bdbb7" Jul 17 18:49:44.086580 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:49:44.086444 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx_odh-ai-gateway-infra(2f699b76-b26f-4579-b271-3095a8ec4194)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx" podUID="2f699b76-b26f-4579-b271-3095a8ec4194" Jul 17 18:49:44.111726 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:44.111699 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7"] Jul 17 18:49:44.241957 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:44.241933 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7" Jul 17 18:49:44.385360 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:44.385286 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa12e8f-8c4f-42ac-87c6-10ec039d824a-maas-api-service-ca\") pod \"cfa12e8f-8c4f-42ac-87c6-10ec039d824a\" (UID: \"cfa12e8f-8c4f-42ac-87c6-10ec039d824a\") " Jul 17 18:49:44.385649 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:44.385629 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa12e8f-8c4f-42ac-87c6-10ec039d824a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cfa12e8f-8c4f-42ac-87c6-10ec039d824a" (UID: "cfa12e8f-8c4f-42ac-87c6-10ec039d824a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:49:44.486206 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:44.486175 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa12e8f-8c4f-42ac-87c6-10ec039d824a-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:49:45.091784 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:45.091752 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7" Jul 17 18:49:45.092290 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:45.091797 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7" event={"ID":"cfa12e8f-8c4f-42ac-87c6-10ec039d824a","Type":"ContainerDied","Data":"bd816dc14e1f3c29e10a37c7d4f07f503eb67e2755459a9f316592c30799b1a8"} Jul 17 18:49:45.092290 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:45.091858 2572 scope.go:117] "RemoveContainer" containerID="1093f40f30cc5adeda421ffb88d3551c0283df8c367ad63bcb551b33620ebf81" Jul 17 18:49:45.093527 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:45.093509 2572 scope.go:117] "RemoveContainer" containerID="2bf7c67e71ff52553fff7054cd8984c3bec360e90e0b72ed9bc48fc7189bdbb7" Jul 17 18:49:45.093697 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:49:45.093679 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx_odh-ai-gateway-infra(2f699b76-b26f-4579-b271-3095a8ec4194)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx" podUID="2f699b76-b26f-4579-b271-3095a8ec4194" Jul 17 18:49:45.128821 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:45.128795 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7"] Jul 17 18:49:45.132588 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:45.132568 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-cdxg7"] Jul 17 18:49:45.225103 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:45.225077 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfa12e8f-8c4f-42ac-87c6-10ec039d824a" path="/var/lib/kubelet/pods/cfa12e8f-8c4f-42ac-87c6-10ec039d824a/volumes" Jul 17 18:49:46.220180 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:46.220151 2572 scope.go:117] "RemoveContainer" containerID="93fa8b31bbed72f40e1ec4096a36d79fb9d20da83d410f91c172903f5688bae3" Jul 17 18:49:47.105437 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:47.105357 2572 generic.go:356] "Generic (PLEG): container finished" podID="0822b07e-213b-4a16-a3d6-f2b51846c5d5" containerID="3a51bca081ac36c482e892b618c662d5078aa6252fc1c3458360991f90b5bb4c" exitCode=7 Jul 17 18:49:47.105437 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:47.105427 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm" event={"ID":"0822b07e-213b-4a16-a3d6-f2b51846c5d5","Type":"ContainerDied","Data":"3a51bca081ac36c482e892b618c662d5078aa6252fc1c3458360991f90b5bb4c"} Jul 17 18:49:47.105646 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:47.105519 2572 scope.go:117] "RemoveContainer" containerID="93fa8b31bbed72f40e1ec4096a36d79fb9d20da83d410f91c172903f5688bae3" Jul 17 18:49:47.105788 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:47.105773 2572 scope.go:117] "RemoveContainer" containerID="3a51bca081ac36c482e892b618c662d5078aa6252fc1c3458360991f90b5bb4c" Jul 17 18:49:47.105995 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:49:47.105976 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm_odh-ai-gateway-infra(0822b07e-213b-4a16-a3d6-f2b51846c5d5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm" podUID="0822b07e-213b-4a16-a3d6-f2b51846c5d5" Jul 17 18:49:48.130349 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:48.130315 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm"] Jul 17 18:49:48.257238 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:48.257215 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm" Jul 17 18:49:48.419643 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:48.419566 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0822b07e-213b-4a16-a3d6-f2b51846c5d5-maas-api-service-ca\") pod \"0822b07e-213b-4a16-a3d6-f2b51846c5d5\" (UID: \"0822b07e-213b-4a16-a3d6-f2b51846c5d5\") " Jul 17 18:49:48.419900 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:48.419877 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0822b07e-213b-4a16-a3d6-f2b51846c5d5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0822b07e-213b-4a16-a3d6-f2b51846c5d5" (UID: "0822b07e-213b-4a16-a3d6-f2b51846c5d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:49:48.520693 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:48.520658 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0822b07e-213b-4a16-a3d6-f2b51846c5d5-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:49:49.119074 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:49.119035 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm" event={"ID":"0822b07e-213b-4a16-a3d6-f2b51846c5d5","Type":"ContainerDied","Data":"50321b9df1890df0ac53578ba3fec3aac10eb7b5ee771f42f735082197ae8bed"} Jul 17 18:49:49.119243 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:49.119106 2572 scope.go:117] "RemoveContainer" containerID="3a51bca081ac36c482e892b618c662d5078aa6252fc1c3458360991f90b5bb4c" Jul 17 18:49:49.119243 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:49.119051 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm" Jul 17 18:49:49.142093 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:49.142065 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm"] Jul 17 18:49:49.145604 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:49.145583 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-t7sxm"] Jul 17 18:49:49.224726 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:49.224688 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0822b07e-213b-4a16-a3d6-f2b51846c5d5" path="/var/lib/kubelet/pods/0822b07e-213b-4a16-a3d6-f2b51846c5d5/volumes" Jul 17 18:49:50.547457 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:49:50.547417 2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa12e8f_8c4f_42ac_87c6_10ec039d824a.slice\": RecentStats: unable to find data in memory cache]" Jul 17 18:49:52.332263 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:49:52.332232 2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa12e8f_8c4f_42ac_87c6_10ec039d824a.slice\": RecentStats: unable to find data in memory cache]" Jul 17 18:49:53.219987 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.219958 2572 scope.go:117] "RemoveContainer" containerID="c50042deefe600803ff83bc6c4d55a1831c3b461887390a900c8b083511f3e15" Jul 17 18:49:53.824273 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.824241 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g"] Jul 17 18:49:53.824645 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.824571 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa12e8f-8c4f-42ac-87c6-10ec039d824a" containerName="revoke-keys" Jul 17 18:49:53.824645 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.824581 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa12e8f-8c4f-42ac-87c6-10ec039d824a" containerName="revoke-keys" Jul 17 18:49:53.824645 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.824589 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa12e8f-8c4f-42ac-87c6-10ec039d824a" containerName="revoke-keys" Jul 17 18:49:53.824645 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.824595 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa12e8f-8c4f-42ac-87c6-10ec039d824a" containerName="revoke-keys" Jul 17 18:49:53.824645 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.824604 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa12e8f-8c4f-42ac-87c6-10ec039d824a" containerName="revoke-keys" Jul 17 18:49:53.824645 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.824610 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa12e8f-8c4f-42ac-87c6-10ec039d824a" containerName="revoke-keys" Jul 17 18:49:53.824645 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.824617 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0822b07e-213b-4a16-a3d6-f2b51846c5d5" containerName="revoke-keys" Jul 17 18:49:53.824645 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.824622 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0822b07e-213b-4a16-a3d6-f2b51846c5d5" containerName="revoke-keys" Jul 17 18:49:53.824645 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.824641 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0822b07e-213b-4a16-a3d6-f2b51846c5d5" containerName="revoke-keys" Jul 17 18:49:53.824645 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.824646 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0822b07e-213b-4a16-a3d6-f2b51846c5d5" containerName="revoke-keys" Jul 17 18:49:53.824942 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.824692 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0822b07e-213b-4a16-a3d6-f2b51846c5d5" containerName="revoke-keys" Jul 17 18:49:53.824942 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.824701 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0822b07e-213b-4a16-a3d6-f2b51846c5d5" containerName="revoke-keys" Jul 17 18:49:53.824942 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.824710 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfa12e8f-8c4f-42ac-87c6-10ec039d824a" containerName="revoke-keys" Jul 17 18:49:53.824942 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.824718 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfa12e8f-8c4f-42ac-87c6-10ec039d824a" containerName="revoke-keys" Jul 17 18:49:53.827658 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.827643 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g" Jul 17 18:49:53.835200 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.835180 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g"] Jul 17 18:49:53.864851 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.864824 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f6573d-0cf2-4dda-ad7d-c1b97692df09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g\" (UID: \"d4f6573d-0cf2-4dda-ad7d-c1b97692df09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g" Jul 17 18:49:53.965768 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.965742 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f6573d-0cf2-4dda-ad7d-c1b97692df09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g\" (UID: \"d4f6573d-0cf2-4dda-ad7d-c1b97692df09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g" Jul 17 18:49:53.966316 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:53.966299 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f6573d-0cf2-4dda-ad7d-c1b97692df09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g\" (UID: \"d4f6573d-0cf2-4dda-ad7d-c1b97692df09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g" Jul 17 18:49:54.140384 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:54.140300 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g" Jul 17 18:49:54.143446 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:54.143400 2572 generic.go:356] "Generic (PLEG): container finished" podID="60bf3694-dcfc-4035-8373-8877f3dff5ea" containerID="8cd0c64c1eda8d2b9a87b5530fc809b4d606f6ca6f6f3243a146eee185d46ba8" exitCode=6 Jul 17 18:49:54.143496 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:54.143458 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx" event={"ID":"60bf3694-dcfc-4035-8373-8877f3dff5ea","Type":"ContainerDied","Data":"8cd0c64c1eda8d2b9a87b5530fc809b4d606f6ca6f6f3243a146eee185d46ba8"} Jul 17 18:49:54.143533 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:54.143509 2572 scope.go:117] "RemoveContainer" containerID="c50042deefe600803ff83bc6c4d55a1831c3b461887390a900c8b083511f3e15" Jul 17 18:49:54.143811 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:54.143784 2572 scope.go:117] "RemoveContainer" containerID="8cd0c64c1eda8d2b9a87b5530fc809b4d606f6ca6f6f3243a146eee185d46ba8" Jul 17 18:49:54.143980 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:49:54.143960 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx_odh-ai-gateway-infra(60bf3694-dcfc-4035-8373-8877f3dff5ea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx" podUID="60bf3694-dcfc-4035-8373-8877f3dff5ea" Jul 17 18:49:54.267970 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:54.267943 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g"] Jul 17 18:49:54.269724 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:49:54.269697 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f6573d_0cf2_4dda_ad7d_c1b97692df09.slice/crio-5a8b24547c61fb09892c641bf21c456436a8967fde5c323c202ea783243b3918 WatchSource:0}: Error finding container 5a8b24547c61fb09892c641bf21c456436a8967fde5c323c202ea783243b3918: Status 404 returned error can't find the container with id 5a8b24547c61fb09892c641bf21c456436a8967fde5c323c202ea783243b3918 Jul 17 18:49:55.150297 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:55.150209 2572 generic.go:356] "Generic (PLEG): container finished" podID="d4f6573d-0cf2-4dda-ad7d-c1b97692df09" containerID="37d72626d745801130f72dff78d842c53402f3b342da61f951d112f689ad4ed5" exitCode=6 Jul 17 18:49:55.150737 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:55.150292 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g" event={"ID":"d4f6573d-0cf2-4dda-ad7d-c1b97692df09","Type":"ContainerDied","Data":"37d72626d745801130f72dff78d842c53402f3b342da61f951d112f689ad4ed5"} Jul 17 18:49:55.150737 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:55.150348 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g" event={"ID":"d4f6573d-0cf2-4dda-ad7d-c1b97692df09","Type":"ContainerStarted","Data":"5a8b24547c61fb09892c641bf21c456436a8967fde5c323c202ea783243b3918"} Jul 17 18:49:55.150737 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:55.150557 2572 scope.go:117] "RemoveContainer" containerID="37d72626d745801130f72dff78d842c53402f3b342da61f951d112f689ad4ed5" Jul 17 18:49:55.171373 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:55.171345 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx"] Jul 17 18:49:55.300912 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:55.300882 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx" Jul 17 18:49:55.376263 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:55.376243 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60bf3694-dcfc-4035-8373-8877f3dff5ea-maas-api-service-ca\") pod \"60bf3694-dcfc-4035-8373-8877f3dff5ea\" (UID: \"60bf3694-dcfc-4035-8373-8877f3dff5ea\") " Jul 17 18:49:55.376577 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:55.376554 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60bf3694-dcfc-4035-8373-8877f3dff5ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60bf3694-dcfc-4035-8373-8877f3dff5ea" (UID: "60bf3694-dcfc-4035-8373-8877f3dff5ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:49:55.476920 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:55.476861 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60bf3694-dcfc-4035-8373-8877f3dff5ea-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:49:56.158648 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:56.158615 2572 generic.go:356] "Generic (PLEG): container finished" podID="d4f6573d-0cf2-4dda-ad7d-c1b97692df09" containerID="3d34a93bbbf58a9865ccef4023cd559168f3a600744d6e97e2cba899a88cb0a3" exitCode=6 Jul 17 18:49:56.159077 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:56.158704 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g" event={"ID":"d4f6573d-0cf2-4dda-ad7d-c1b97692df09","Type":"ContainerDied","Data":"3d34a93bbbf58a9865ccef4023cd559168f3a600744d6e97e2cba899a88cb0a3"} Jul 17 18:49:56.159077 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:56.158758 2572 scope.go:117] "RemoveContainer" containerID="37d72626d745801130f72dff78d842c53402f3b342da61f951d112f689ad4ed5" Jul 17 18:49:56.159077 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:56.158912 2572 scope.go:117] "RemoveContainer" containerID="3d34a93bbbf58a9865ccef4023cd559168f3a600744d6e97e2cba899a88cb0a3" Jul 17 18:49:56.159266 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:49:56.159153 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g_odh-ai-gateway-infra(d4f6573d-0cf2-4dda-ad7d-c1b97692df09)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g" podUID="d4f6573d-0cf2-4dda-ad7d-c1b97692df09" Jul 17 18:49:56.159929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:56.159902 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx" Jul 17 18:49:56.160034 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:56.159940 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx" event={"ID":"60bf3694-dcfc-4035-8373-8877f3dff5ea","Type":"ContainerDied","Data":"f25725d24da433313e3954184be8431243abb09455bef3bee8646c4186fbb218"} Jul 17 18:49:56.169480 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:56.169456 2572 scope.go:117] "RemoveContainer" containerID="8cd0c64c1eda8d2b9a87b5530fc809b4d606f6ca6f6f3243a146eee185d46ba8" Jul 17 18:49:56.189670 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:56.189646 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx"] Jul 17 18:49:56.193895 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:56.193875 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-9tlmx"] Jul 17 18:49:56.219639 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:56.219622 2572 scope.go:117] "RemoveContainer" containerID="2bf7c67e71ff52553fff7054cd8984c3bec360e90e0b72ed9bc48fc7189bdbb7" Jul 17 18:49:57.170271 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:57.170194 2572 generic.go:356] "Generic (PLEG): container finished" podID="2f699b76-b26f-4579-b271-3095a8ec4194" containerID="f932682e8b0e29237715cb789174789f89374d88eb632bc82de1e816933e3980" exitCode=6 Jul 17 18:49:57.170731 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:57.170268 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx" event={"ID":"2f699b76-b26f-4579-b271-3095a8ec4194","Type":"ContainerDied","Data":"f932682e8b0e29237715cb789174789f89374d88eb632bc82de1e816933e3980"} Jul 17 18:49:57.170731 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:57.170320 2572 scope.go:117] "RemoveContainer" containerID="2bf7c67e71ff52553fff7054cd8984c3bec360e90e0b72ed9bc48fc7189bdbb7" Jul 17 18:49:57.170731 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:57.170607 2572 scope.go:117] "RemoveContainer" containerID="f932682e8b0e29237715cb789174789f89374d88eb632bc82de1e816933e3980" Jul 17 18:49:57.170873 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:49:57.170840 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx_odh-ai-gateway-infra(2f699b76-b26f-4579-b271-3095a8ec4194)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx" podUID="2f699b76-b26f-4579-b271-3095a8ec4194" Jul 17 18:49:57.173871 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:57.172739 2572 scope.go:117] "RemoveContainer" containerID="3d34a93bbbf58a9865ccef4023cd559168f3a600744d6e97e2cba899a88cb0a3" Jul 17 18:49:57.173871 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:49:57.173130 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g_odh-ai-gateway-infra(d4f6573d-0cf2-4dda-ad7d-c1b97692df09)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g" podUID="d4f6573d-0cf2-4dda-ad7d-c1b97692df09" Jul 17 18:49:57.224119 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:57.224086 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60bf3694-dcfc-4035-8373-8877f3dff5ea" path="/var/lib/kubelet/pods/60bf3694-dcfc-4035-8373-8877f3dff5ea/volumes" Jul 17 18:49:58.197972 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:58.197941 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx"] Jul 17 18:49:58.325461 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:58.325425 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx" Jul 17 18:49:58.401352 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:58.401325 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f699b76-b26f-4579-b271-3095a8ec4194-maas-api-service-ca\") pod \"2f699b76-b26f-4579-b271-3095a8ec4194\" (UID: \"2f699b76-b26f-4579-b271-3095a8ec4194\") " Jul 17 18:49:58.401681 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:58.401661 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f699b76-b26f-4579-b271-3095a8ec4194-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2f699b76-b26f-4579-b271-3095a8ec4194" (UID: "2f699b76-b26f-4579-b271-3095a8ec4194"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:49:58.502843 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:58.502772 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f699b76-b26f-4579-b271-3095a8ec4194-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:49:59.184166 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:59.184135 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx" Jul 17 18:49:59.184464 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:59.184163 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx" event={"ID":"2f699b76-b26f-4579-b271-3095a8ec4194","Type":"ContainerDied","Data":"f2906624d9678cc6893b40c87c0d53c3fb7e1c242f0f4aa8b3459740b4dbfe43"} Jul 17 18:49:59.184464 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:59.184233 2572 scope.go:117] "RemoveContainer" containerID="f932682e8b0e29237715cb789174789f89374d88eb632bc82de1e816933e3980" Jul 17 18:49:59.204581 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:59.204544 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx"] Jul 17 18:49:59.210604 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:59.210580 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-9bjkx"] Jul 17 18:49:59.223927 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:49:59.223904 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f699b76-b26f-4579-b271-3095a8ec4194" path="/var/lib/kubelet/pods/2f699b76-b26f-4579-b271-3095a8ec4194/volumes" Jul 17 18:50:02.367286 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:02.367256 2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa12e8f_8c4f_42ac_87c6_10ec039d824a.slice\": RecentStats: unable to find data in memory cache]" Jul 17 18:50:05.523896 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:05.523859 2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa12e8f_8c4f_42ac_87c6_10ec039d824a.slice\": RecentStats: unable to find data in memory cache]" Jul 17 18:50:10.219414 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:10.219383 2572 scope.go:117] "RemoveContainer" containerID="3d34a93bbbf58a9865ccef4023cd559168f3a600744d6e97e2cba899a88cb0a3" Jul 17 18:50:11.247867 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:11.247774 2572 generic.go:356] "Generic (PLEG): container finished" podID="d4f6573d-0cf2-4dda-ad7d-c1b97692df09" containerID="1bb3835e42ecfd3ca1a66cbf763128f94437fb5b369dbf9bb7227173ed1c55d1" exitCode=6 Jul 17 18:50:11.247867 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:11.247816 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g" event={"ID":"d4f6573d-0cf2-4dda-ad7d-c1b97692df09","Type":"ContainerDied","Data":"1bb3835e42ecfd3ca1a66cbf763128f94437fb5b369dbf9bb7227173ed1c55d1"} Jul 17 18:50:11.248351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:11.247899 2572 scope.go:117] "RemoveContainer" containerID="3d34a93bbbf58a9865ccef4023cd559168f3a600744d6e97e2cba899a88cb0a3" Jul 17 18:50:11.248351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:11.248173 2572 scope.go:117] "RemoveContainer" containerID="1bb3835e42ecfd3ca1a66cbf763128f94437fb5b369dbf9bb7227173ed1c55d1" Jul 17 18:50:11.248429 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:11.248391 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g_odh-ai-gateway-infra(d4f6573d-0cf2-4dda-ad7d-c1b97692df09)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g" podUID="d4f6573d-0cf2-4dda-ad7d-c1b97692df09" Jul 17 18:50:12.272105 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:12.272073 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g"] Jul 17 18:50:12.406485 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:12.406461 2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa12e8f_8c4f_42ac_87c6_10ec039d824a.slice\": RecentStats: unable to find data in memory cache]" Jul 17 18:50:12.407313 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:12.407288 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g" Jul 17 18:50:12.519433 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:12.519404 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f6573d-0cf2-4dda-ad7d-c1b97692df09-maas-api-service-ca\") pod \"d4f6573d-0cf2-4dda-ad7d-c1b97692df09\" (UID: \"d4f6573d-0cf2-4dda-ad7d-c1b97692df09\") " Jul 17 18:50:12.519745 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:12.519721 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f6573d-0cf2-4dda-ad7d-c1b97692df09-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4f6573d-0cf2-4dda-ad7d-c1b97692df09" (UID: "d4f6573d-0cf2-4dda-ad7d-c1b97692df09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:50:12.620495 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:12.620457 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f6573d-0cf2-4dda-ad7d-c1b97692df09-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:50:13.259078 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:13.259048 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g" Jul 17 18:50:13.259078 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:13.259057 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g" event={"ID":"d4f6573d-0cf2-4dda-ad7d-c1b97692df09","Type":"ContainerDied","Data":"5a8b24547c61fb09892c641bf21c456436a8967fde5c323c202ea783243b3918"} Jul 17 18:50:13.259310 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:13.259120 2572 scope.go:117] "RemoveContainer" containerID="1bb3835e42ecfd3ca1a66cbf763128f94437fb5b369dbf9bb7227173ed1c55d1" Jul 17 18:50:13.275355 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:13.275328 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g"] Jul 17 18:50:13.283425 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:13.283395 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ftj2g"] Jul 17 18:50:15.224435 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.224402 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4f6573d-0cf2-4dda-ad7d-c1b97692df09" path="/var/lib/kubelet/pods/d4f6573d-0cf2-4dda-ad7d-c1b97692df09/volumes" Jul 17 18:50:15.696810 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.696779 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x"] Jul 17 18:50:15.697119 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697104 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f699b76-b26f-4579-b271-3095a8ec4194" containerName="revoke-keys" Jul 17 18:50:15.697171 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697121 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f699b76-b26f-4579-b271-3095a8ec4194" containerName="revoke-keys" Jul 17 18:50:15.697171 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697132 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f6573d-0cf2-4dda-ad7d-c1b97692df09" containerName="revoke-keys" Jul 17 18:50:15.697171 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697140 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f6573d-0cf2-4dda-ad7d-c1b97692df09" containerName="revoke-keys" Jul 17 18:50:15.697171 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697156 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f699b76-b26f-4579-b271-3095a8ec4194" containerName="revoke-keys" Jul 17 18:50:15.697171 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697164 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f699b76-b26f-4579-b271-3095a8ec4194" containerName="revoke-keys" Jul 17 18:50:15.697171 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697170 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0822b07e-213b-4a16-a3d6-f2b51846c5d5" containerName="revoke-keys" Jul 17 18:50:15.697364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697176 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0822b07e-213b-4a16-a3d6-f2b51846c5d5" containerName="revoke-keys" Jul 17 18:50:15.697364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697181 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60bf3694-dcfc-4035-8373-8877f3dff5ea" containerName="revoke-keys" Jul 17 18:50:15.697364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697187 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="60bf3694-dcfc-4035-8373-8877f3dff5ea" containerName="revoke-keys" Jul 17 18:50:15.697364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697194 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60bf3694-dcfc-4035-8373-8877f3dff5ea" containerName="revoke-keys" Jul 17 18:50:15.697364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697199 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="60bf3694-dcfc-4035-8373-8877f3dff5ea" containerName="revoke-keys" Jul 17 18:50:15.697364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697214 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f6573d-0cf2-4dda-ad7d-c1b97692df09" containerName="revoke-keys" Jul 17 18:50:15.697364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697220 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f6573d-0cf2-4dda-ad7d-c1b97692df09" containerName="revoke-keys" Jul 17 18:50:15.697364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697267 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f6573d-0cf2-4dda-ad7d-c1b97692df09" containerName="revoke-keys" Jul 17 18:50:15.697364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697273 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="60bf3694-dcfc-4035-8373-8877f3dff5ea" containerName="revoke-keys" Jul 17 18:50:15.697364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697279 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f699b76-b26f-4579-b271-3095a8ec4194" containerName="revoke-keys" Jul 17 18:50:15.697364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697285 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="60bf3694-dcfc-4035-8373-8877f3dff5ea" containerName="revoke-keys" Jul 17 18:50:15.697364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697291 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfa12e8f-8c4f-42ac-87c6-10ec039d824a" containerName="revoke-keys" Jul 17 18:50:15.697364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697299 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0822b07e-213b-4a16-a3d6-f2b51846c5d5" containerName="revoke-keys" Jul 17 18:50:15.697364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697304 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f699b76-b26f-4579-b271-3095a8ec4194" containerName="revoke-keys" Jul 17 18:50:15.697364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697309 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f6573d-0cf2-4dda-ad7d-c1b97692df09" containerName="revoke-keys" Jul 17 18:50:15.697364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.697314 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f6573d-0cf2-4dda-ad7d-c1b97692df09" containerName="revoke-keys" Jul 17 18:50:15.701622 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.701605 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x" Jul 17 18:50:15.704113 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.704100 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:50:15.704361 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.704345 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:50:15.708959 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.708502 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x"] Jul 17 18:50:15.746929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.746897 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x\" (UID: \"c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x" Jul 17 18:50:15.847298 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.847265 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x\" (UID: \"c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x" Jul 17 18:50:15.847875 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:15.847855 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x\" (UID: \"c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x" Jul 17 18:50:16.012627 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:16.012547 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x" Jul 17 18:50:16.135917 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:16.135891 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x"] Jul 17 18:50:16.137502 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:50:16.137463 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2fe0ac7_8db0_43a5_adf7_0a9cb53a616e.slice/crio-8e6ad16fc36daf939993aaa0ea850dc67914366644d2c8f6d16900ebc0e2eafb WatchSource:0}: Error finding container 8e6ad16fc36daf939993aaa0ea850dc67914366644d2c8f6d16900ebc0e2eafb: Status 404 returned error can't find the container with id 8e6ad16fc36daf939993aaa0ea850dc67914366644d2c8f6d16900ebc0e2eafb Jul 17 18:50:16.272919 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:16.272847 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x" event={"ID":"c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e","Type":"ContainerStarted","Data":"8e6ad16fc36daf939993aaa0ea850dc67914366644d2c8f6d16900ebc0e2eafb"} Jul 17 18:50:17.277833 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:17.277799 2572 generic.go:356] "Generic (PLEG): container finished" podID="c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e" containerID="53efdaf58fac68ad65d469bdc8a2c6ee0d88b9500330e62674178ea0938c5e77" exitCode=6 Jul 17 18:50:17.278242 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:17.277864 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x" event={"ID":"c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e","Type":"ContainerDied","Data":"53efdaf58fac68ad65d469bdc8a2c6ee0d88b9500330e62674178ea0938c5e77"} Jul 17 18:50:17.278242 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:17.278090 2572 scope.go:117] "RemoveContainer" containerID="53efdaf58fac68ad65d469bdc8a2c6ee0d88b9500330e62674178ea0938c5e77" Jul 17 18:50:18.284190 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:18.284156 2572 generic.go:356] "Generic (PLEG): container finished" podID="c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e" containerID="c401c31bc643390aee407ef1cc91cb487ee8eef1e8d876708783ece2ab2d1879" exitCode=6 Jul 17 18:50:18.284582 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:18.284210 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x" event={"ID":"c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e","Type":"ContainerDied","Data":"c401c31bc643390aee407ef1cc91cb487ee8eef1e8d876708783ece2ab2d1879"} Jul 17 18:50:18.284582 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:18.284259 2572 scope.go:117] "RemoveContainer" containerID="53efdaf58fac68ad65d469bdc8a2c6ee0d88b9500330e62674178ea0938c5e77" Jul 17 18:50:18.284582 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:18.284496 2572 scope.go:117] "RemoveContainer" containerID="c401c31bc643390aee407ef1cc91cb487ee8eef1e8d876708783ece2ab2d1879" Jul 17 18:50:18.284751 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:18.284708 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x_odh-ai-gateway-infra(c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x" podUID="c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e" Jul 17 18:50:19.289689 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:19.289663 2572 scope.go:117] "RemoveContainer" containerID="c401c31bc643390aee407ef1cc91cb487ee8eef1e8d876708783ece2ab2d1879" Jul 17 18:50:19.290081 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:19.289832 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x_odh-ai-gateway-infra(c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x" podUID="c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e" Jul 17 18:50:20.087762 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:20.087726 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-m675m"] Jul 17 18:50:20.088049 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:20.088037 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f6573d-0cf2-4dda-ad7d-c1b97692df09" containerName="revoke-keys" Jul 17 18:50:20.088107 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:20.088050 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f6573d-0cf2-4dda-ad7d-c1b97692df09" containerName="revoke-keys" Jul 17 18:50:20.088107 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:20.088062 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f699b76-b26f-4579-b271-3095a8ec4194" containerName="revoke-keys" Jul 17 18:50:20.088107 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:20.088068 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f699b76-b26f-4579-b271-3095a8ec4194" containerName="revoke-keys" Jul 17 18:50:20.088107 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:20.088077 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60bf3694-dcfc-4035-8373-8877f3dff5ea" containerName="revoke-keys" Jul 17 18:50:20.088107 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:20.088082 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="60bf3694-dcfc-4035-8373-8877f3dff5ea" containerName="revoke-keys" Jul 17 18:50:20.088266 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:20.088126 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f699b76-b26f-4579-b271-3095a8ec4194" containerName="revoke-keys" Jul 17 18:50:20.088266 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:20.088134 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="60bf3694-dcfc-4035-8373-8877f3dff5ea" containerName="revoke-keys" Jul 17 18:50:20.091854 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:20.091839 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-m675m" Jul 17 18:50:20.098486 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:20.098459 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-m675m"] Jul 17 18:50:20.183704 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:20.183680 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73a8ea8-927b-4242-8bbf-2cfebcabea93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a03eb5-m675m\" (UID: \"d73a8ea8-927b-4242-8bbf-2cfebcabea93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-m675m" Jul 17 18:50:20.284615 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:20.284585 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73a8ea8-927b-4242-8bbf-2cfebcabea93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a03eb5-m675m\" (UID: \"d73a8ea8-927b-4242-8bbf-2cfebcabea93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-m675m" Jul 17 18:50:20.285214 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:20.285196 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73a8ea8-927b-4242-8bbf-2cfebcabea93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a03eb5-m675m\" (UID: \"d73a8ea8-927b-4242-8bbf-2cfebcabea93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-m675m" Jul 17 18:50:20.402854 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:20.402779 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-m675m" Jul 17 18:50:20.526312 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:20.526290 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-m675m"] Jul 17 18:50:20.528462 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:50:20.528440 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73a8ea8_927b_4242_8bbf_2cfebcabea93.slice/crio-e286a27ef6142d44c1945674df9af2f241791ef5597302422dffbd75e6ee38a4 WatchSource:0}: Error finding container e286a27ef6142d44c1945674df9af2f241791ef5597302422dffbd75e6ee38a4: Status 404 returned error can't find the container with id e286a27ef6142d44c1945674df9af2f241791ef5597302422dffbd75e6ee38a4 Jul 17 18:50:20.559918 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:20.559894 2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa12e8f_8c4f_42ac_87c6_10ec039d824a.slice\": RecentStats: unable to find data in memory cache]" Jul 17 18:50:21.299431 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:21.299400 2572 generic.go:356] "Generic (PLEG): container finished" podID="d73a8ea8-927b-4242-8bbf-2cfebcabea93" containerID="e55489e1fd86ef92b53b5a86dc846dc0de31b87177fc77f7fc362388db87b61c" exitCode=0 Jul 17 18:50:21.299557 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:21.299470 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-m675m" event={"ID":"d73a8ea8-927b-4242-8bbf-2cfebcabea93","Type":"ContainerDied","Data":"e55489e1fd86ef92b53b5a86dc846dc0de31b87177fc77f7fc362388db87b61c"} Jul 17 18:50:21.299557 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:21.299522 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-m675m" event={"ID":"d73a8ea8-927b-4242-8bbf-2cfebcabea93","Type":"ContainerStarted","Data":"e286a27ef6142d44c1945674df9af2f241791ef5597302422dffbd75e6ee38a4"} Jul 17 18:50:22.414843 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:22.414813 2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa12e8f_8c4f_42ac_87c6_10ec039d824a.slice\": RecentStats: unable to find data in memory cache]" Jul 17 18:50:22.436872 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:22.436849 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-m675m" Jul 17 18:50:22.505028 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:22.504988 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73a8ea8-927b-4242-8bbf-2cfebcabea93-maas-api-service-ca\") pod \"d73a8ea8-927b-4242-8bbf-2cfebcabea93\" (UID: \"d73a8ea8-927b-4242-8bbf-2cfebcabea93\") " Jul 17 18:50:22.505321 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:22.505298 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73a8ea8-927b-4242-8bbf-2cfebcabea93-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d73a8ea8-927b-4242-8bbf-2cfebcabea93" (UID: "d73a8ea8-927b-4242-8bbf-2cfebcabea93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:50:22.605889 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:22.605814 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d73a8ea8-927b-4242-8bbf-2cfebcabea93-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:50:23.309378 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:23.309348 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-m675m" Jul 17 18:50:23.309549 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:23.309375 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a03eb5-m675m" event={"ID":"d73a8ea8-927b-4242-8bbf-2cfebcabea93","Type":"ContainerDied","Data":"e286a27ef6142d44c1945674df9af2f241791ef5597302422dffbd75e6ee38a4"} Jul 17 18:50:23.309549 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:23.309424 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e286a27ef6142d44c1945674df9af2f241791ef5597302422dffbd75e6ee38a4" Jul 17 18:50:23.521226 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:23.521142 2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa12e8f_8c4f_42ac_87c6_10ec039d824a.slice\": RecentStats: unable to find data in memory cache]" Jul 17 18:50:23.521600 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:23.521297 2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa12e8f_8c4f_42ac_87c6_10ec039d824a.slice\": RecentStats: unable to find data in memory cache]" Jul 17 18:50:27.827113 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:27.827078 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg"] Jul 17 18:50:27.827597 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:27.827525 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d73a8ea8-927b-4242-8bbf-2cfebcabea93" containerName="revoke-keys" Jul 17 18:50:27.827597 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:27.827543 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d73a8ea8-927b-4242-8bbf-2cfebcabea93" containerName="revoke-keys" Jul 17 18:50:27.827735 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:27.827620 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d73a8ea8-927b-4242-8bbf-2cfebcabea93" containerName="revoke-keys" Jul 17 18:50:27.830583 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:27.830561 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg" Jul 17 18:50:27.837510 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:27.837490 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg"] Jul 17 18:50:27.954391 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:27.954358 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43ddb2e-f642-41f4-bb4f-b5c9ba62decc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg\" (UID: \"a43ddb2e-f642-41f4-bb4f-b5c9ba62decc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg" Jul 17 18:50:28.055648 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:28.055615 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43ddb2e-f642-41f4-bb4f-b5c9ba62decc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg\" (UID: \"a43ddb2e-f642-41f4-bb4f-b5c9ba62decc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg" Jul 17 18:50:28.056240 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:28.056221 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43ddb2e-f642-41f4-bb4f-b5c9ba62decc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg\" (UID: \"a43ddb2e-f642-41f4-bb4f-b5c9ba62decc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg" Jul 17 18:50:28.141684 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:28.141620 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg" Jul 17 18:50:28.270037 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:28.269984 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg"] Jul 17 18:50:28.271938 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:50:28.271904 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43ddb2e_f642_41f4_bb4f_b5c9ba62decc.slice/crio-332fc11e4a207da9822c2708e8c829ff8c6df7f73e5b5dd4b5083a2f65725029 WatchSource:0}: Error finding container 332fc11e4a207da9822c2708e8c829ff8c6df7f73e5b5dd4b5083a2f65725029: Status 404 returned error can't find the container with id 332fc11e4a207da9822c2708e8c829ff8c6df7f73e5b5dd4b5083a2f65725029 Jul 17 18:50:28.331309 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:28.331277 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg" event={"ID":"a43ddb2e-f642-41f4-bb4f-b5c9ba62decc","Type":"ContainerStarted","Data":"332fc11e4a207da9822c2708e8c829ff8c6df7f73e5b5dd4b5083a2f65725029"} Jul 17 18:50:29.337318 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:29.337285 2572 generic.go:356] "Generic (PLEG): container finished" podID="a43ddb2e-f642-41f4-bb4f-b5c9ba62decc" containerID="22b274400058e409b7a6f5bcd3d478bb6627dd17ac06aa549c2d55a4aafb3980" exitCode=6 Jul 17 18:50:29.337704 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:29.337371 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg" event={"ID":"a43ddb2e-f642-41f4-bb4f-b5c9ba62decc","Type":"ContainerDied","Data":"22b274400058e409b7a6f5bcd3d478bb6627dd17ac06aa549c2d55a4aafb3980"} Jul 17 18:50:29.337704 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:29.337554 2572 scope.go:117] "RemoveContainer" containerID="22b274400058e409b7a6f5bcd3d478bb6627dd17ac06aa549c2d55a4aafb3980" Jul 17 18:50:30.344670 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:30.344642 2572 generic.go:356] "Generic (PLEG): container finished" podID="a43ddb2e-f642-41f4-bb4f-b5c9ba62decc" containerID="48da099c938cbe5e73d9303579969bbf429cac7c66f0c81bbbdf9ea97f729ad4" exitCode=6 Jul 17 18:50:30.345227 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:30.344678 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg" event={"ID":"a43ddb2e-f642-41f4-bb4f-b5c9ba62decc","Type":"ContainerDied","Data":"48da099c938cbe5e73d9303579969bbf429cac7c66f0c81bbbdf9ea97f729ad4"} Jul 17 18:50:30.345227 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:30.344735 2572 scope.go:117] "RemoveContainer" containerID="22b274400058e409b7a6f5bcd3d478bb6627dd17ac06aa549c2d55a4aafb3980" Jul 17 18:50:30.345227 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:30.344891 2572 scope.go:117] "RemoveContainer" containerID="48da099c938cbe5e73d9303579969bbf429cac7c66f0c81bbbdf9ea97f729ad4" Jul 17 18:50:30.345227 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:30.345142 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg_odh-ai-gateway-infra(a43ddb2e-f642-41f4-bb4f-b5c9ba62decc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg" podUID="a43ddb2e-f642-41f4-bb4f-b5c9ba62decc" Jul 17 18:50:31.314914 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:31.314883 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l"] Jul 17 18:50:31.319335 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:31.319319 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l" Jul 17 18:50:31.324669 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:31.324644 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l"] Jul 17 18:50:31.351036 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:31.351001 2572 scope.go:117] "RemoveContainer" containerID="48da099c938cbe5e73d9303579969bbf429cac7c66f0c81bbbdf9ea97f729ad4" Jul 17 18:50:31.351412 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:31.351254 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg_odh-ai-gateway-infra(a43ddb2e-f642-41f4-bb4f-b5c9ba62decc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg" podUID="a43ddb2e-f642-41f4-bb4f-b5c9ba62decc" Jul 17 18:50:31.483750 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:31.483716 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44605f08-9cee-4e07-b3b1-f80ea540ccc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l\" (UID: \"44605f08-9cee-4e07-b3b1-f80ea540ccc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l" Jul 17 18:50:31.584426 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:31.584346 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44605f08-9cee-4e07-b3b1-f80ea540ccc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l\" (UID: \"44605f08-9cee-4e07-b3b1-f80ea540ccc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l" Jul 17 18:50:31.585030 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:31.584981 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44605f08-9cee-4e07-b3b1-f80ea540ccc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l\" (UID: \"44605f08-9cee-4e07-b3b1-f80ea540ccc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l" Jul 17 18:50:31.630928 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:31.630899 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l" Jul 17 18:50:31.756401 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:31.756378 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l"] Jul 17 18:50:31.758666 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:50:31.758635 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44605f08_9cee_4e07_b3b1_f80ea540ccc0.slice/crio-182946dad96551033e43f3353f2682462813a8c4dc4bdfcc84b105d82c9521de WatchSource:0}: Error finding container 182946dad96551033e43f3353f2682462813a8c4dc4bdfcc84b105d82c9521de: Status 404 returned error can't find the container with id 182946dad96551033e43f3353f2682462813a8c4dc4bdfcc84b105d82c9521de Jul 17 18:50:32.363207 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:32.363172 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l" event={"ID":"44605f08-9cee-4e07-b3b1-f80ea540ccc0","Type":"ContainerStarted","Data":"73f49afbe2fe806f4a9580e98b52f786998add135fcb2211f030a2f6f7b0e293"} Jul 17 18:50:32.363652 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:32.363243 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l" event={"ID":"44605f08-9cee-4e07-b3b1-f80ea540ccc0","Type":"ContainerStarted","Data":"182946dad96551033e43f3353f2682462813a8c4dc4bdfcc84b105d82c9521de"} Jul 17 18:50:32.378087 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:32.378041 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l" podStartSLOduration=1.378028942 podStartE2EDuration="1.378028942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:50:32.377026978 +0000 UTC m=+3227.786454608" watchObservedRunningTime="2026-07-17 18:50:32.378028942 +0000 UTC m=+3227.787456538" Jul 17 18:50:32.451353 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:32.451327 2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa12e8f_8c4f_42ac_87c6_10ec039d824a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44605f08_9cee_4e07_b3b1_f80ea540ccc0.slice/crio-73f49afbe2fe806f4a9580e98b52f786998add135fcb2211f030a2f6f7b0e293.scope\": RecentStats: unable to find data in memory cache]" Jul 17 18:50:33.369461 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:33.369428 2572 generic.go:356] "Generic (PLEG): container finished" podID="44605f08-9cee-4e07-b3b1-f80ea540ccc0" containerID="73f49afbe2fe806f4a9580e98b52f786998add135fcb2211f030a2f6f7b0e293" exitCode=6 Jul 17 18:50:33.369846 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:33.369518 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l" event={"ID":"44605f08-9cee-4e07-b3b1-f80ea540ccc0","Type":"ContainerDied","Data":"73f49afbe2fe806f4a9580e98b52f786998add135fcb2211f030a2f6f7b0e293"} Jul 17 18:50:33.369846 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:33.369689 2572 scope.go:117] "RemoveContainer" containerID="73f49afbe2fe806f4a9580e98b52f786998add135fcb2211f030a2f6f7b0e293" Jul 17 18:50:34.220306 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:34.220235 2572 scope.go:117] "RemoveContainer" containerID="c401c31bc643390aee407ef1cc91cb487ee8eef1e8d876708783ece2ab2d1879" Jul 17 18:50:34.376366 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:34.376326 2572 generic.go:356] "Generic (PLEG): container finished" podID="44605f08-9cee-4e07-b3b1-f80ea540ccc0" containerID="19163229dce488bd1da05242d46497fb1bcb63d731f55ae7b74f22ed5fe8b897" exitCode=6 Jul 17 18:50:34.376841 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:34.376393 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l" event={"ID":"44605f08-9cee-4e07-b3b1-f80ea540ccc0","Type":"ContainerDied","Data":"19163229dce488bd1da05242d46497fb1bcb63d731f55ae7b74f22ed5fe8b897"} Jul 17 18:50:34.376841 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:34.376445 2572 scope.go:117] "RemoveContainer" containerID="73f49afbe2fe806f4a9580e98b52f786998add135fcb2211f030a2f6f7b0e293" Jul 17 18:50:34.376841 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:34.376627 2572 scope.go:117] "RemoveContainer" containerID="19163229dce488bd1da05242d46497fb1bcb63d731f55ae7b74f22ed5fe8b897" Jul 17 18:50:34.376995 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:34.376889 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l_odh-ai-gateway-infra(44605f08-9cee-4e07-b3b1-f80ea540ccc0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l" podUID="44605f08-9cee-4e07-b3b1-f80ea540ccc0" Jul 17 18:50:35.382455 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:35.382423 2572 generic.go:356] "Generic (PLEG): container finished" podID="c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e" containerID="6f03c25f5cc19abbb83ea2a67ea8609ee65c99a1f8e8201a5826791382823628" exitCode=6 Jul 17 18:50:35.382918 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:35.382501 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x" event={"ID":"c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e","Type":"ContainerDied","Data":"6f03c25f5cc19abbb83ea2a67ea8609ee65c99a1f8e8201a5826791382823628"} Jul 17 18:50:35.382918 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:35.382565 2572 scope.go:117] "RemoveContainer" containerID="c401c31bc643390aee407ef1cc91cb487ee8eef1e8d876708783ece2ab2d1879" Jul 17 18:50:35.382918 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:35.382883 2572 scope.go:117] "RemoveContainer" containerID="6f03c25f5cc19abbb83ea2a67ea8609ee65c99a1f8e8201a5826791382823628" Jul 17 18:50:35.383148 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:35.383126 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x_odh-ai-gateway-infra(c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x" podUID="c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e" Jul 17 18:50:35.384235 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:35.384219 2572 scope.go:117] "RemoveContainer" containerID="19163229dce488bd1da05242d46497fb1bcb63d731f55ae7b74f22ed5fe8b897" Jul 17 18:50:35.384427 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:35.384408 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l_odh-ai-gateway-infra(44605f08-9cee-4e07-b3b1-f80ea540ccc0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l" podUID="44605f08-9cee-4e07-b3b1-f80ea540ccc0" Jul 17 18:50:35.517339 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:35.517305 2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa12e8f_8c4f_42ac_87c6_10ec039d824a.slice\": RecentStats: unable to find data in memory cache]" Jul 17 18:50:36.409960 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:36.409928 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x"] Jul 17 18:50:36.536672 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:36.536646 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x" Jul 17 18:50:36.629301 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:36.629266 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e-maas-api-service-ca\") pod \"c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e\" (UID: \"c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e\") " Jul 17 18:50:36.629649 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:36.629626 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e" (UID: "c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:50:36.730503 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:36.730428 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:50:37.395878 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:37.395855 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x" Jul 17 18:50:37.396082 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:37.395892 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x" event={"ID":"c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e","Type":"ContainerDied","Data":"8e6ad16fc36daf939993aaa0ea850dc67914366644d2c8f6d16900ebc0e2eafb"} Jul 17 18:50:37.396082 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:37.395951 2572 scope.go:117] "RemoveContainer" containerID="6f03c25f5cc19abbb83ea2a67ea8609ee65c99a1f8e8201a5826791382823628" Jul 17 18:50:37.411868 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:37.411843 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x"] Jul 17 18:50:37.415314 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:37.415293 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-hjv6x"] Jul 17 18:50:39.225218 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:39.225185 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e" path="/var/lib/kubelet/pods/c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e/volumes" Jul 17 18:50:42.459134 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:42.459100 2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa12e8f_8c4f_42ac_87c6_10ec039d824a.slice\": RecentStats: unable to find data in memory cache]" Jul 17 18:50:43.219760 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:43.219725 2572 scope.go:117] "RemoveContainer" containerID="48da099c938cbe5e73d9303579969bbf429cac7c66f0c81bbbdf9ea97f729ad4" Jul 17 18:50:43.862957 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:43.862932 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56"] Jul 17 18:50:43.863347 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:43.863245 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e" containerName="revoke-keys" Jul 17 18:50:43.863347 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:43.863255 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e" containerName="revoke-keys" Jul 17 18:50:43.863347 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:43.863278 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e" containerName="revoke-keys" Jul 17 18:50:43.863347 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:43.863284 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e" containerName="revoke-keys" Jul 17 18:50:43.863347 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:43.863336 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e" containerName="revoke-keys" Jul 17 18:50:43.863347 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:43.863343 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e" containerName="revoke-keys" Jul 17 18:50:43.863347 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:43.863350 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e" containerName="revoke-keys" Jul 17 18:50:43.866118 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:43.866099 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56" Jul 17 18:50:43.872848 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:43.872825 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56"] Jul 17 18:50:43.885861 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:43.885838 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59757297-a5a8-47b1-9f7f-594ddf856cf5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-4bq56\" (UID: \"59757297-a5a8-47b1-9f7f-594ddf856cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56" Jul 17 18:50:43.986461 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:43.986438 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59757297-a5a8-47b1-9f7f-594ddf856cf5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-4bq56\" (UID: \"59757297-a5a8-47b1-9f7f-594ddf856cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56" Jul 17 18:50:43.986986 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:43.986968 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59757297-a5a8-47b1-9f7f-594ddf856cf5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-4bq56\" (UID: \"59757297-a5a8-47b1-9f7f-594ddf856cf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56" Jul 17 18:50:44.217613 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:44.217529 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56" Jul 17 18:50:44.343273 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:44.343246 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56"] Jul 17 18:50:44.345299 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:50:44.345269 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59757297_a5a8_47b1_9f7f_594ddf856cf5.slice/crio-f66f5cc53d431297c60f3a50ff6ff8f2d37aaf813f5f1ab047ad9f1ef575cac9 WatchSource:0}: Error finding container f66f5cc53d431297c60f3a50ff6ff8f2d37aaf813f5f1ab047ad9f1ef575cac9: Status 404 returned error can't find the container with id f66f5cc53d431297c60f3a50ff6ff8f2d37aaf813f5f1ab047ad9f1ef575cac9 Jul 17 18:50:44.428402 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:44.428378 2572 generic.go:356] "Generic (PLEG): container finished" podID="a43ddb2e-f642-41f4-bb4f-b5c9ba62decc" containerID="4478f68318b66acbaa0b7fffd8708aa5320e25185917cbcd373d0394bef70428" exitCode=6 Jul 17 18:50:44.428539 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:44.428447 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg" event={"ID":"a43ddb2e-f642-41f4-bb4f-b5c9ba62decc","Type":"ContainerDied","Data":"4478f68318b66acbaa0b7fffd8708aa5320e25185917cbcd373d0394bef70428"} Jul 17 18:50:44.428539 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:44.428507 2572 scope.go:117] "RemoveContainer" containerID="48da099c938cbe5e73d9303579969bbf429cac7c66f0c81bbbdf9ea97f729ad4" Jul 17 18:50:44.428845 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:44.428823 2572 scope.go:117] "RemoveContainer" containerID="4478f68318b66acbaa0b7fffd8708aa5320e25185917cbcd373d0394bef70428" Jul 17 18:50:44.429109 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:44.429088 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg_odh-ai-gateway-infra(a43ddb2e-f642-41f4-bb4f-b5c9ba62decc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg" podUID="a43ddb2e-f642-41f4-bb4f-b5c9ba62decc" Jul 17 18:50:44.429747 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:44.429726 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56" event={"ID":"59757297-a5a8-47b1-9f7f-594ddf856cf5","Type":"ContainerStarted","Data":"f66f5cc53d431297c60f3a50ff6ff8f2d37aaf813f5f1ab047ad9f1ef575cac9"} Jul 17 18:50:45.437858 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:45.437766 2572 generic.go:356] "Generic (PLEG): container finished" podID="59757297-a5a8-47b1-9f7f-594ddf856cf5" containerID="3102baddeb6f10d4ec3a2af94a8d2af144ab26340381a1d329a5d74f7e1eadf9" exitCode=6 Jul 17 18:50:45.437858 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:45.437847 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56" event={"ID":"59757297-a5a8-47b1-9f7f-594ddf856cf5","Type":"ContainerDied","Data":"3102baddeb6f10d4ec3a2af94a8d2af144ab26340381a1d329a5d74f7e1eadf9"} Jul 17 18:50:45.438312 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:45.438045 2572 scope.go:117] "RemoveContainer" containerID="3102baddeb6f10d4ec3a2af94a8d2af144ab26340381a1d329a5d74f7e1eadf9" Jul 17 18:50:45.452371 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:45.452347 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg"] Jul 17 18:50:45.581836 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:45.581813 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg" Jul 17 18:50:45.598836 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:45.598637 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43ddb2e-f642-41f4-bb4f-b5c9ba62decc-maas-api-service-ca\") pod \"a43ddb2e-f642-41f4-bb4f-b5c9ba62decc\" (UID: \"a43ddb2e-f642-41f4-bb4f-b5c9ba62decc\") " Jul 17 18:50:45.599070 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:45.599046 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a43ddb2e-f642-41f4-bb4f-b5c9ba62decc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a43ddb2e-f642-41f4-bb4f-b5c9ba62decc" (UID: "a43ddb2e-f642-41f4-bb4f-b5c9ba62decc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:50:45.699380 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:45.699322 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43ddb2e-f642-41f4-bb4f-b5c9ba62decc-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:50:46.250943 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.250911 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9de544-xcxtw"] Jul 17 18:50:46.251233 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.251221 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43ddb2e-f642-41f4-bb4f-b5c9ba62decc" containerName="revoke-keys" Jul 17 18:50:46.251282 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.251235 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43ddb2e-f642-41f4-bb4f-b5c9ba62decc" containerName="revoke-keys" Jul 17 18:50:46.251282 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.251242 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e" containerName="revoke-keys" Jul 17 18:50:46.251282 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.251247 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2fe0ac7-8db0-43a5-adf7-0a9cb53a616e" containerName="revoke-keys" Jul 17 18:50:46.251282 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.251255 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43ddb2e-f642-41f4-bb4f-b5c9ba62decc" containerName="revoke-keys" Jul 17 18:50:46.251282 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.251261 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43ddb2e-f642-41f4-bb4f-b5c9ba62decc" containerName="revoke-keys" Jul 17 18:50:46.251443 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.251322 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a43ddb2e-f642-41f4-bb4f-b5c9ba62decc" containerName="revoke-keys" Jul 17 18:50:46.251443 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.251331 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a43ddb2e-f642-41f4-bb4f-b5c9ba62decc" containerName="revoke-keys" Jul 17 18:50:46.251443 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.251338 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a43ddb2e-f642-41f4-bb4f-b5c9ba62decc" containerName="revoke-keys" Jul 17 18:50:46.254195 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.254177 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9de544-xcxtw" Jul 17 18:50:46.261990 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.261948 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9de544-xcxtw"] Jul 17 18:50:46.304965 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.304942 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66455439-835c-4a6b-bc36-c67b9df80148-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9de544-xcxtw\" (UID: \"66455439-835c-4a6b-bc36-c67b9df80148\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9de544-xcxtw" Jul 17 18:50:46.406380 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.406310 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66455439-835c-4a6b-bc36-c67b9df80148-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9de544-xcxtw\" (UID: \"66455439-835c-4a6b-bc36-c67b9df80148\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9de544-xcxtw" Jul 17 18:50:46.406876 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.406859 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66455439-835c-4a6b-bc36-c67b9df80148-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9de544-xcxtw\" (UID: \"66455439-835c-4a6b-bc36-c67b9df80148\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9de544-xcxtw" Jul 17 18:50:46.443317 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.443293 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg" Jul 17 18:50:46.443718 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.443296 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg" event={"ID":"a43ddb2e-f642-41f4-bb4f-b5c9ba62decc","Type":"ContainerDied","Data":"332fc11e4a207da9822c2708e8c829ff8c6df7f73e5b5dd4b5083a2f65725029"} Jul 17 18:50:46.443718 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.443439 2572 scope.go:117] "RemoveContainer" containerID="4478f68318b66acbaa0b7fffd8708aa5320e25185917cbcd373d0394bef70428" Jul 17 18:50:46.445235 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.445212 2572 generic.go:356] "Generic (PLEG): container finished" podID="59757297-a5a8-47b1-9f7f-594ddf856cf5" containerID="fcea7b30e2422f621bb604d33e5f3dc2251dfb190387f83703177f08f3afb555" exitCode=6 Jul 17 18:50:46.445362 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.445279 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56" event={"ID":"59757297-a5a8-47b1-9f7f-594ddf856cf5","Type":"ContainerDied","Data":"fcea7b30e2422f621bb604d33e5f3dc2251dfb190387f83703177f08f3afb555"} Jul 17 18:50:46.445496 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.445482 2572 scope.go:117] "RemoveContainer" containerID="fcea7b30e2422f621bb604d33e5f3dc2251dfb190387f83703177f08f3afb555" Jul 17 18:50:46.445698 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:46.445679 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-4bq56_odh-ai-gateway-infra(59757297-a5a8-47b1-9f7f-594ddf856cf5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56" podUID="59757297-a5a8-47b1-9f7f-594ddf856cf5" Jul 17 18:50:46.452023 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.451990 2572 scope.go:117] "RemoveContainer" containerID="3102baddeb6f10d4ec3a2af94a8d2af144ab26340381a1d329a5d74f7e1eadf9" Jul 17 18:50:46.475121 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.475096 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg"] Jul 17 18:50:46.479262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.479238 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-h9xfg"] Jul 17 18:50:46.564939 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.564918 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9de544-xcxtw" Jul 17 18:50:46.690781 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:46.690730 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9de544-xcxtw"] Jul 17 18:50:46.693204 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:50:46.693177 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66455439_835c_4a6b_bc36_c67b9df80148.slice/crio-da1fb672a8c383dc45b1a1a68147cb271e2ca4f9bdb4e0d4783233219271d5eb WatchSource:0}: Error finding container da1fb672a8c383dc45b1a1a68147cb271e2ca4f9bdb4e0d4783233219271d5eb: Status 404 returned error can't find the container with id da1fb672a8c383dc45b1a1a68147cb271e2ca4f9bdb4e0d4783233219271d5eb Jul 17 18:50:47.225505 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:47.225466 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a43ddb2e-f642-41f4-bb4f-b5c9ba62decc" path="/var/lib/kubelet/pods/a43ddb2e-f642-41f4-bb4f-b5c9ba62decc/volumes" Jul 17 18:50:47.451159 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:47.451132 2572 generic.go:356] "Generic (PLEG): container finished" podID="66455439-835c-4a6b-bc36-c67b9df80148" containerID="ab3b5fb1a3a788d804eca07263fd970fd574c3d10e8b3cdbddad05fb85ad1e15" exitCode=0 Jul 17 18:50:47.451518 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:47.451210 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9de544-xcxtw" event={"ID":"66455439-835c-4a6b-bc36-c67b9df80148","Type":"ContainerDied","Data":"ab3b5fb1a3a788d804eca07263fd970fd574c3d10e8b3cdbddad05fb85ad1e15"} Jul 17 18:50:47.451518 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:47.451266 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9de544-xcxtw" event={"ID":"66455439-835c-4a6b-bc36-c67b9df80148","Type":"ContainerStarted","Data":"da1fb672a8c383dc45b1a1a68147cb271e2ca4f9bdb4e0d4783233219271d5eb"} Jul 17 18:50:47.454229 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:47.454210 2572 scope.go:117] "RemoveContainer" containerID="fcea7b30e2422f621bb604d33e5f3dc2251dfb190387f83703177f08f3afb555" Jul 17 18:50:47.454457 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:47.454436 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-4bq56_odh-ai-gateway-infra(59757297-a5a8-47b1-9f7f-594ddf856cf5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56" podUID="59757297-a5a8-47b1-9f7f-594ddf856cf5" Jul 17 18:50:48.569207 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:48.569184 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9de544-xcxtw" Jul 17 18:50:48.623216 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:48.623188 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66455439-835c-4a6b-bc36-c67b9df80148-maas-api-service-ca\") pod \"66455439-835c-4a6b-bc36-c67b9df80148\" (UID: \"66455439-835c-4a6b-bc36-c67b9df80148\") " Jul 17 18:50:48.623550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:48.623527 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66455439-835c-4a6b-bc36-c67b9df80148-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "66455439-835c-4a6b-bc36-c67b9df80148" (UID: "66455439-835c-4a6b-bc36-c67b9df80148"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:50:48.724469 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:48.724388 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66455439-835c-4a6b-bc36-c67b9df80148-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:50:49.219799 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:49.219773 2572 scope.go:117] "RemoveContainer" containerID="19163229dce488bd1da05242d46497fb1bcb63d731f55ae7b74f22ed5fe8b897" Jul 17 18:50:49.463581 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:49.463556 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9de544-xcxtw" Jul 17 18:50:49.463722 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:49.463596 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9de544-xcxtw" event={"ID":"66455439-835c-4a6b-bc36-c67b9df80148","Type":"ContainerDied","Data":"da1fb672a8c383dc45b1a1a68147cb271e2ca4f9bdb4e0d4783233219271d5eb"} Jul 17 18:50:49.463722 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:49.463664 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da1fb672a8c383dc45b1a1a68147cb271e2ca4f9bdb4e0d4783233219271d5eb" Jul 17 18:50:50.470352 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:50.470316 2572 generic.go:356] "Generic (PLEG): container finished" podID="44605f08-9cee-4e07-b3b1-f80ea540ccc0" containerID="4bbf6b012117f98ac15c843790b8dc694220d1e19eb394e76f9e8c731626f9b7" exitCode=6 Jul 17 18:50:50.470779 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:50.470391 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l" event={"ID":"44605f08-9cee-4e07-b3b1-f80ea540ccc0","Type":"ContainerDied","Data":"4bbf6b012117f98ac15c843790b8dc694220d1e19eb394e76f9e8c731626f9b7"} Jul 17 18:50:50.470779 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:50.470455 2572 scope.go:117] "RemoveContainer" containerID="19163229dce488bd1da05242d46497fb1bcb63d731f55ae7b74f22ed5fe8b897" Jul 17 18:50:50.470779 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:50.470712 2572 scope.go:117] "RemoveContainer" containerID="4bbf6b012117f98ac15c843790b8dc694220d1e19eb394e76f9e8c731626f9b7" Jul 17 18:50:50.470934 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:50:50.470915 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l_odh-ai-gateway-infra(44605f08-9cee-4e07-b3b1-f80ea540ccc0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l" podUID="44605f08-9cee-4e07-b3b1-f80ea540ccc0" Jul 17 18:50:51.494988 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:51.494956 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l"] Jul 17 18:50:51.628957 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:51.628927 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l" Jul 17 18:50:51.650567 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:51.650542 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44605f08-9cee-4e07-b3b1-f80ea540ccc0-maas-api-service-ca\") pod \"44605f08-9cee-4e07-b3b1-f80ea540ccc0\" (UID: \"44605f08-9cee-4e07-b3b1-f80ea540ccc0\") " Jul 17 18:50:51.650860 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:51.650841 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44605f08-9cee-4e07-b3b1-f80ea540ccc0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "44605f08-9cee-4e07-b3b1-f80ea540ccc0" (UID: "44605f08-9cee-4e07-b3b1-f80ea540ccc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:50:51.751397 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:51.751324 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44605f08-9cee-4e07-b3b1-f80ea540ccc0-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:50:52.481137 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:52.481103 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l" event={"ID":"44605f08-9cee-4e07-b3b1-f80ea540ccc0","Type":"ContainerDied","Data":"182946dad96551033e43f3353f2682462813a8c4dc4bdfcc84b105d82c9521de"} Jul 17 18:50:52.481137 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:52.481124 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l" Jul 17 18:50:52.481389 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:52.481165 2572 scope.go:117] "RemoveContainer" containerID="4bbf6b012117f98ac15c843790b8dc694220d1e19eb394e76f9e8c731626f9b7" Jul 17 18:50:52.508295 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:52.508270 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l"] Jul 17 18:50:52.512677 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:52.512653 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-bjf4l"] Jul 17 18:50:53.224350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:53.224313 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44605f08-9cee-4e07-b3b1-f80ea540ccc0" path="/var/lib/kubelet/pods/44605f08-9cee-4e07-b3b1-f80ea540ccc0/volumes" Jul 17 18:50:56.694724 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:56.694688 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9de544-29738565-t76pw"] Jul 17 18:50:56.696802 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:56.696780 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-9de544-29738565-t76pw"] Jul 17 18:50:57.225267 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:57.225229 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c1800f7-6b69-4fd1-bd6c-8e96b0cf3551" path="/var/lib/kubelet/pods/4c1800f7-6b69-4fd1-bd6c-8e96b0cf3551/volumes" Jul 17 18:50:59.219990 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:50:59.219960 2572 scope.go:117] "RemoveContainer" containerID="fcea7b30e2422f621bb604d33e5f3dc2251dfb190387f83703177f08f3afb555" Jul 17 18:51:00.517768 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:00.517736 2572 generic.go:356] "Generic (PLEG): container finished" podID="59757297-a5a8-47b1-9f7f-594ddf856cf5" containerID="e68fcf4b2506d71604b4bd413ba001c9a3923b930a43474e7ef9c7852e26d666" exitCode=6 Jul 17 18:51:00.518207 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:00.517799 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56" event={"ID":"59757297-a5a8-47b1-9f7f-594ddf856cf5","Type":"ContainerDied","Data":"e68fcf4b2506d71604b4bd413ba001c9a3923b930a43474e7ef9c7852e26d666"} Jul 17 18:51:00.518207 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:00.517846 2572 scope.go:117] "RemoveContainer" containerID="fcea7b30e2422f621bb604d33e5f3dc2251dfb190387f83703177f08f3afb555" Jul 17 18:51:00.518207 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:00.518176 2572 scope.go:117] "RemoveContainer" containerID="e68fcf4b2506d71604b4bd413ba001c9a3923b930a43474e7ef9c7852e26d666" Jul 17 18:51:00.518398 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:51:00.518380 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-4bq56_odh-ai-gateway-infra(59757297-a5a8-47b1-9f7f-594ddf856cf5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56" podUID="59757297-a5a8-47b1-9f7f-594ddf856cf5" Jul 17 18:51:01.543596 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:01.543564 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56"] Jul 17 18:51:01.670157 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:01.670134 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56" Jul 17 18:51:01.734997 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:01.734964 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59757297-a5a8-47b1-9f7f-594ddf856cf5-maas-api-service-ca\") pod \"59757297-a5a8-47b1-9f7f-594ddf856cf5\" (UID: \"59757297-a5a8-47b1-9f7f-594ddf856cf5\") " Jul 17 18:51:01.735355 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:01.735330 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59757297-a5a8-47b1-9f7f-594ddf856cf5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59757297-a5a8-47b1-9f7f-594ddf856cf5" (UID: "59757297-a5a8-47b1-9f7f-594ddf856cf5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:51:01.835651 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:01.835564 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59757297-a5a8-47b1-9f7f-594ddf856cf5-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:51:02.529647 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:02.529619 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56" Jul 17 18:51:02.529647 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:02.529617 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56" event={"ID":"59757297-a5a8-47b1-9f7f-594ddf856cf5","Type":"ContainerDied","Data":"f66f5cc53d431297c60f3a50ff6ff8f2d37aaf813f5f1ab047ad9f1ef575cac9"} Jul 17 18:51:02.529883 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:02.529674 2572 scope.go:117] "RemoveContainer" containerID="e68fcf4b2506d71604b4bd413ba001c9a3923b930a43474e7ef9c7852e26d666" Jul 17 18:51:02.549309 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:02.549285 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56"] Jul 17 18:51:02.554479 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:02.554459 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-4bq56"] Jul 17 18:51:03.224552 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:03.224519 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59757297-a5a8-47b1-9f7f-594ddf856cf5" path="/var/lib/kubelet/pods/59757297-a5a8-47b1-9f7f-594ddf856cf5/volumes" Jul 17 18:51:15.740733 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.740696 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf"] Jul 17 18:51:15.741142 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741033 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59757297-a5a8-47b1-9f7f-594ddf856cf5" containerName="revoke-keys" Jul 17 18:51:15.741142 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741045 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="59757297-a5a8-47b1-9f7f-594ddf856cf5" containerName="revoke-keys" Jul 17 18:51:15.741142 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741059 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44605f08-9cee-4e07-b3b1-f80ea540ccc0" containerName="revoke-keys" Jul 17 18:51:15.741142 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741064 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="44605f08-9cee-4e07-b3b1-f80ea540ccc0" containerName="revoke-keys" Jul 17 18:51:15.741142 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741070 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44605f08-9cee-4e07-b3b1-f80ea540ccc0" containerName="revoke-keys" Jul 17 18:51:15.741142 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741075 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="44605f08-9cee-4e07-b3b1-f80ea540ccc0" containerName="revoke-keys" Jul 17 18:51:15.741142 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741081 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43ddb2e-f642-41f4-bb4f-b5c9ba62decc" containerName="revoke-keys" Jul 17 18:51:15.741142 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741086 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43ddb2e-f642-41f4-bb4f-b5c9ba62decc" containerName="revoke-keys" Jul 17 18:51:15.741142 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741095 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66455439-835c-4a6b-bc36-c67b9df80148" containerName="revoke-keys" Jul 17 18:51:15.741142 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741100 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="66455439-835c-4a6b-bc36-c67b9df80148" containerName="revoke-keys" Jul 17 18:51:15.741142 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741108 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59757297-a5a8-47b1-9f7f-594ddf856cf5" containerName="revoke-keys" Jul 17 18:51:15.741142 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741113 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="59757297-a5a8-47b1-9f7f-594ddf856cf5" containerName="revoke-keys" Jul 17 18:51:15.741142 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741125 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44605f08-9cee-4e07-b3b1-f80ea540ccc0" containerName="revoke-keys" Jul 17 18:51:15.741142 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741130 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="44605f08-9cee-4e07-b3b1-f80ea540ccc0" containerName="revoke-keys" Jul 17 18:51:15.741590 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741176 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="44605f08-9cee-4e07-b3b1-f80ea540ccc0" containerName="revoke-keys" Jul 17 18:51:15.741590 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741183 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="59757297-a5a8-47b1-9f7f-594ddf856cf5" containerName="revoke-keys" Jul 17 18:51:15.741590 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741188 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="44605f08-9cee-4e07-b3b1-f80ea540ccc0" containerName="revoke-keys" Jul 17 18:51:15.741590 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741194 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="66455439-835c-4a6b-bc36-c67b9df80148" containerName="revoke-keys" Jul 17 18:51:15.741590 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741201 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="59757297-a5a8-47b1-9f7f-594ddf856cf5" containerName="revoke-keys" Jul 17 18:51:15.741590 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.741206 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="59757297-a5a8-47b1-9f7f-594ddf856cf5" containerName="revoke-keys" Jul 17 18:51:15.745302 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.745286 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf" Jul 17 18:51:15.747723 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.747693 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:51:15.747842 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.747827 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:51:15.751659 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.751635 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf"] Jul 17 18:51:15.859347 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.859321 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26e94ca7-3c76-4dc2-a5d7-91965c8a96c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf\" (UID: \"26e94ca7-3c76-4dc2-a5d7-91965c8a96c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf" Jul 17 18:51:15.960546 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.960508 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26e94ca7-3c76-4dc2-a5d7-91965c8a96c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf\" (UID: \"26e94ca7-3c76-4dc2-a5d7-91965c8a96c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf" Jul 17 18:51:15.961096 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:15.961078 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26e94ca7-3c76-4dc2-a5d7-91965c8a96c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf\" (UID: \"26e94ca7-3c76-4dc2-a5d7-91965c8a96c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf" Jul 17 18:51:16.056979 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:16.056951 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf" Jul 17 18:51:16.180279 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:16.180220 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf"] Jul 17 18:51:16.182977 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:51:16.182953 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e94ca7_3c76_4dc2_a5d7_91965c8a96c2.slice/crio-ca8f758bdda59aa17e71e65a8ec344e74ed51c99cf3992b3708edb5488a89d18 WatchSource:0}: Error finding container ca8f758bdda59aa17e71e65a8ec344e74ed51c99cf3992b3708edb5488a89d18: Status 404 returned error can't find the container with id ca8f758bdda59aa17e71e65a8ec344e74ed51c99cf3992b3708edb5488a89d18 Jul 17 18:51:16.590902 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:16.590865 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf" event={"ID":"26e94ca7-3c76-4dc2-a5d7-91965c8a96c2","Type":"ContainerStarted","Data":"2b2a3804d0d331ad321741ce5226c3d6fd186485193631f482d908c246ce7f95"} Jul 17 18:51:16.591124 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:16.590924 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf" event={"ID":"26e94ca7-3c76-4dc2-a5d7-91965c8a96c2","Type":"ContainerStarted","Data":"ca8f758bdda59aa17e71e65a8ec344e74ed51c99cf3992b3708edb5488a89d18"} Jul 17 18:51:16.612126 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:16.612034 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf" podStartSLOduration=1.6120038829999999 podStartE2EDuration="1.612003883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:51:16.611209255 +0000 UTC m=+3272.020636861" watchObservedRunningTime="2026-07-17 18:51:16.612003883 +0000 UTC m=+3272.021431489" Jul 17 18:51:17.597255 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:17.597223 2572 generic.go:356] "Generic (PLEG): container finished" podID="26e94ca7-3c76-4dc2-a5d7-91965c8a96c2" containerID="2b2a3804d0d331ad321741ce5226c3d6fd186485193631f482d908c246ce7f95" exitCode=6 Jul 17 18:51:17.597730 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:17.597285 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf" event={"ID":"26e94ca7-3c76-4dc2-a5d7-91965c8a96c2","Type":"ContainerDied","Data":"2b2a3804d0d331ad321741ce5226c3d6fd186485193631f482d908c246ce7f95"} Jul 17 18:51:17.597730 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:17.597510 2572 scope.go:117] "RemoveContainer" containerID="2b2a3804d0d331ad321741ce5226c3d6fd186485193631f482d908c246ce7f95" Jul 17 18:51:18.603430 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:18.603397 2572 generic.go:356] "Generic (PLEG): container finished" podID="26e94ca7-3c76-4dc2-a5d7-91965c8a96c2" containerID="686de59c57c8bf1ecd9d25c8705e5d297863d08737fc5186e126d119c05dbb58" exitCode=6 Jul 17 18:51:18.603794 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:18.603478 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf" event={"ID":"26e94ca7-3c76-4dc2-a5d7-91965c8a96c2","Type":"ContainerDied","Data":"686de59c57c8bf1ecd9d25c8705e5d297863d08737fc5186e126d119c05dbb58"} Jul 17 18:51:18.603794 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:18.603533 2572 scope.go:117] "RemoveContainer" containerID="2b2a3804d0d331ad321741ce5226c3d6fd186485193631f482d908c246ce7f95" Jul 17 18:51:18.603794 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:18.603759 2572 scope.go:117] "RemoveContainer" containerID="686de59c57c8bf1ecd9d25c8705e5d297863d08737fc5186e126d119c05dbb58" Jul 17 18:51:18.603997 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:51:18.603976 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf_odh-ai-gateway-infra(26e94ca7-3c76-4dc2-a5d7-91965c8a96c2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf" podUID="26e94ca7-3c76-4dc2-a5d7-91965c8a96c2" Jul 17 18:51:19.610358 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:19.610335 2572 scope.go:117] "RemoveContainer" containerID="686de59c57c8bf1ecd9d25c8705e5d297863d08737fc5186e126d119c05dbb58" Jul 17 18:51:19.610706 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:51:19.610544 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf_odh-ai-gateway-infra(26e94ca7-3c76-4dc2-a5d7-91965c8a96c2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf" podUID="26e94ca7-3c76-4dc2-a5d7-91965c8a96c2" Jul 17 18:51:31.361255 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:31.361213 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz"] Jul 17 18:51:31.361740 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:31.361646 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59757297-a5a8-47b1-9f7f-594ddf856cf5" containerName="revoke-keys" Jul 17 18:51:31.361740 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:31.361664 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="59757297-a5a8-47b1-9f7f-594ddf856cf5" containerName="revoke-keys" Jul 17 18:51:31.361859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:31.361757 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="44605f08-9cee-4e07-b3b1-f80ea540ccc0" containerName="revoke-keys" Jul 17 18:51:31.364678 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:31.364656 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz" Jul 17 18:51:31.371201 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:31.371180 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz"] Jul 17 18:51:31.394218 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:31.394195 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a93079ad-108a-4dec-89cb-c4c3887460c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-747tz\" (UID: \"a93079ad-108a-4dec-89cb-c4c3887460c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz" Jul 17 18:51:31.495197 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:31.495165 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a93079ad-108a-4dec-89cb-c4c3887460c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-747tz\" (UID: \"a93079ad-108a-4dec-89cb-c4c3887460c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz" Jul 17 18:51:31.495812 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:31.495793 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a93079ad-108a-4dec-89cb-c4c3887460c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-747tz\" (UID: \"a93079ad-108a-4dec-89cb-c4c3887460c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz" Jul 17 18:51:31.676151 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:31.676071 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz" Jul 17 18:51:32.007109 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:32.007081 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz"] Jul 17 18:51:32.009552 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:51:32.009530 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93079ad_108a_4dec_89cb_c4c3887460c5.slice/crio-be4aabe8d4d14dbb8af80052e59befac806554f7b6b799bf4de70ec4a3fe162b WatchSource:0}: Error finding container be4aabe8d4d14dbb8af80052e59befac806554f7b6b799bf4de70ec4a3fe162b: Status 404 returned error can't find the container with id be4aabe8d4d14dbb8af80052e59befac806554f7b6b799bf4de70ec4a3fe162b Jul 17 18:51:32.676410 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:32.676381 2572 generic.go:356] "Generic (PLEG): container finished" podID="a93079ad-108a-4dec-89cb-c4c3887460c5" containerID="ecdff3e25573d4e3b4964de4536ac0db8d2208ba32cc30bb37ddc0ebc4239a8b" exitCode=6 Jul 17 18:51:32.676791 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:32.676475 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz" event={"ID":"a93079ad-108a-4dec-89cb-c4c3887460c5","Type":"ContainerDied","Data":"ecdff3e25573d4e3b4964de4536ac0db8d2208ba32cc30bb37ddc0ebc4239a8b"} Jul 17 18:51:32.676791 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:32.676541 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz" event={"ID":"a93079ad-108a-4dec-89cb-c4c3887460c5","Type":"ContainerStarted","Data":"be4aabe8d4d14dbb8af80052e59befac806554f7b6b799bf4de70ec4a3fe162b"} Jul 17 18:51:32.676791 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:32.676624 2572 scope.go:117] "RemoveContainer" containerID="ecdff3e25573d4e3b4964de4536ac0db8d2208ba32cc30bb37ddc0ebc4239a8b" Jul 17 18:51:33.230846 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:33.230773 2572 scope.go:117] "RemoveContainer" containerID="686de59c57c8bf1ecd9d25c8705e5d297863d08737fc5186e126d119c05dbb58" Jul 17 18:51:33.686118 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:33.686082 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf" event={"ID":"26e94ca7-3c76-4dc2-a5d7-91965c8a96c2","Type":"ContainerStarted","Data":"fdef9ef5c15b113316108de7ba3dad6c3062905ea46fe69c96c9f15f967e937a"} Jul 17 18:51:33.687852 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:33.687825 2572 generic.go:356] "Generic (PLEG): container finished" podID="a93079ad-108a-4dec-89cb-c4c3887460c5" containerID="74f19e8fdfd02e193ec8e20e49be7bc07913175f2152abf2ff8a450ede2f72a5" exitCode=6 Jul 17 18:51:33.687992 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:33.687888 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz" event={"ID":"a93079ad-108a-4dec-89cb-c4c3887460c5","Type":"ContainerDied","Data":"74f19e8fdfd02e193ec8e20e49be7bc07913175f2152abf2ff8a450ede2f72a5"} Jul 17 18:51:33.687992 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:33.687930 2572 scope.go:117] "RemoveContainer" containerID="ecdff3e25573d4e3b4964de4536ac0db8d2208ba32cc30bb37ddc0ebc4239a8b" Jul 17 18:51:33.688125 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:33.688054 2572 scope.go:117] "RemoveContainer" containerID="74f19e8fdfd02e193ec8e20e49be7bc07913175f2152abf2ff8a450ede2f72a5" Jul 17 18:51:33.688285 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:51:33.688266 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-747tz_odh-ai-gateway-infra(a93079ad-108a-4dec-89cb-c4c3887460c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz" podUID="a93079ad-108a-4dec-89cb-c4c3887460c5" Jul 17 18:51:33.913414 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:33.913386 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz"] Jul 17 18:51:33.916422 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:33.916407 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz" Jul 17 18:51:33.924618 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:33.924592 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz"] Jul 17 18:51:34.016102 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:34.016025 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f547776d-5ed6-476a-a1c0-91c47dad660f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-f44tz\" (UID: \"f547776d-5ed6-476a-a1c0-91c47dad660f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz" Jul 17 18:51:34.116645 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:34.116612 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f547776d-5ed6-476a-a1c0-91c47dad660f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-f44tz\" (UID: \"f547776d-5ed6-476a-a1c0-91c47dad660f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz" Jul 17 18:51:34.117236 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:34.117218 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f547776d-5ed6-476a-a1c0-91c47dad660f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-f44tz\" (UID: \"f547776d-5ed6-476a-a1c0-91c47dad660f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz" Jul 17 18:51:34.227039 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:34.226997 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz" Jul 17 18:51:34.351275 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:34.351250 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz"] Jul 17 18:51:34.353364 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:51:34.353335 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf547776d_5ed6_476a_a1c0_91c47dad660f.slice/crio-26657b80932a42ba214a71427867b42281cac180333a7d0670ca5f8197f7264a WatchSource:0}: Error finding container 26657b80932a42ba214a71427867b42281cac180333a7d0670ca5f8197f7264a: Status 404 returned error can't find the container with id 26657b80932a42ba214a71427867b42281cac180333a7d0670ca5f8197f7264a Jul 17 18:51:34.693588 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:34.693561 2572 generic.go:356] "Generic (PLEG): container finished" podID="26e94ca7-3c76-4dc2-a5d7-91965c8a96c2" containerID="fdef9ef5c15b113316108de7ba3dad6c3062905ea46fe69c96c9f15f967e937a" exitCode=6 Jul 17 18:51:34.693989 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:34.693637 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf" event={"ID":"26e94ca7-3c76-4dc2-a5d7-91965c8a96c2","Type":"ContainerDied","Data":"fdef9ef5c15b113316108de7ba3dad6c3062905ea46fe69c96c9f15f967e937a"} Jul 17 18:51:34.693989 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:34.693711 2572 scope.go:117] "RemoveContainer" containerID="686de59c57c8bf1ecd9d25c8705e5d297863d08737fc5186e126d119c05dbb58" Jul 17 18:51:34.694143 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:34.693998 2572 scope.go:117] "RemoveContainer" containerID="fdef9ef5c15b113316108de7ba3dad6c3062905ea46fe69c96c9f15f967e937a" Jul 17 18:51:34.694260 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:51:34.694241 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf_odh-ai-gateway-infra(26e94ca7-3c76-4dc2-a5d7-91965c8a96c2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf" podUID="26e94ca7-3c76-4dc2-a5d7-91965c8a96c2" Jul 17 18:51:34.695259 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:34.695234 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz" event={"ID":"f547776d-5ed6-476a-a1c0-91c47dad660f","Type":"ContainerStarted","Data":"04d4181ea312e82e1f7224a7951504521b63c5bdbaef764313eb89e5b6c903b8"} Jul 17 18:51:34.695361 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:34.695283 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz" event={"ID":"f547776d-5ed6-476a-a1c0-91c47dad660f","Type":"ContainerStarted","Data":"26657b80932a42ba214a71427867b42281cac180333a7d0670ca5f8197f7264a"} Jul 17 18:51:34.696877 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:34.696858 2572 scope.go:117] "RemoveContainer" containerID="74f19e8fdfd02e193ec8e20e49be7bc07913175f2152abf2ff8a450ede2f72a5" Jul 17 18:51:34.697034 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:51:34.696998 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-747tz_odh-ai-gateway-infra(a93079ad-108a-4dec-89cb-c4c3887460c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz" podUID="a93079ad-108a-4dec-89cb-c4c3887460c5" Jul 17 18:51:34.717443 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:34.717417 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf"] Jul 17 18:51:34.740175 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:34.740134 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz" podStartSLOduration=1.740122794 podStartE2EDuration="1.740122794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:51:34.738703958 +0000 UTC m=+3290.148131566" watchObservedRunningTime="2026-07-17 18:51:34.740122794 +0000 UTC m=+3290.149550400" Jul 17 18:51:35.703686 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:35.703658 2572 generic.go:356] "Generic (PLEG): container finished" podID="f547776d-5ed6-476a-a1c0-91c47dad660f" containerID="04d4181ea312e82e1f7224a7951504521b63c5bdbaef764313eb89e5b6c903b8" exitCode=6 Jul 17 18:51:35.704086 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:35.703710 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz" event={"ID":"f547776d-5ed6-476a-a1c0-91c47dad660f","Type":"ContainerDied","Data":"04d4181ea312e82e1f7224a7951504521b63c5bdbaef764313eb89e5b6c903b8"} Jul 17 18:51:35.704086 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:35.703965 2572 scope.go:117] "RemoveContainer" containerID="04d4181ea312e82e1f7224a7951504521b63c5bdbaef764313eb89e5b6c903b8" Jul 17 18:51:35.835602 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:35.835579 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf" Jul 17 18:51:35.932880 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:35.932854 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26e94ca7-3c76-4dc2-a5d7-91965c8a96c2-maas-api-service-ca\") pod \"26e94ca7-3c76-4dc2-a5d7-91965c8a96c2\" (UID: \"26e94ca7-3c76-4dc2-a5d7-91965c8a96c2\") " Jul 17 18:51:35.933249 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:35.933220 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e94ca7-3c76-4dc2-a5d7-91965c8a96c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "26e94ca7-3c76-4dc2-a5d7-91965c8a96c2" (UID: "26e94ca7-3c76-4dc2-a5d7-91965c8a96c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:51:36.033485 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:36.033459 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26e94ca7-3c76-4dc2-a5d7-91965c8a96c2-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:51:36.709802 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:36.709706 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf" Jul 17 18:51:36.709802 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:36.709706 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf" event={"ID":"26e94ca7-3c76-4dc2-a5d7-91965c8a96c2","Type":"ContainerDied","Data":"ca8f758bdda59aa17e71e65a8ec344e74ed51c99cf3992b3708edb5488a89d18"} Jul 17 18:51:36.710297 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:36.709856 2572 scope.go:117] "RemoveContainer" containerID="fdef9ef5c15b113316108de7ba3dad6c3062905ea46fe69c96c9f15f967e937a" Jul 17 18:51:36.711700 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:36.711680 2572 generic.go:356] "Generic (PLEG): container finished" podID="f547776d-5ed6-476a-a1c0-91c47dad660f" containerID="81e5f0325bef1ae80099baafad2830e9b9f222a22f184e3fbc7b1f1adb102263" exitCode=6 Jul 17 18:51:36.711824 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:36.711726 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz" event={"ID":"f547776d-5ed6-476a-a1c0-91c47dad660f","Type":"ContainerDied","Data":"81e5f0325bef1ae80099baafad2830e9b9f222a22f184e3fbc7b1f1adb102263"} Jul 17 18:51:36.711989 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:36.711972 2572 scope.go:117] "RemoveContainer" containerID="81e5f0325bef1ae80099baafad2830e9b9f222a22f184e3fbc7b1f1adb102263" Jul 17 18:51:36.712236 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:51:36.712208 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-f44tz_odh-ai-gateway-infra(f547776d-5ed6-476a-a1c0-91c47dad660f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz" podUID="f547776d-5ed6-476a-a1c0-91c47dad660f" Jul 17 18:51:36.718529 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:36.718513 2572 scope.go:117] "RemoveContainer" containerID="04d4181ea312e82e1f7224a7951504521b63c5bdbaef764313eb89e5b6c903b8" Jul 17 18:51:36.745679 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:36.745654 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf"] Jul 17 18:51:36.749124 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:36.749104 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8lfsf"] Jul 17 18:51:37.224919 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:37.224875 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26e94ca7-3c76-4dc2-a5d7-91965c8a96c2" path="/var/lib/kubelet/pods/26e94ca7-3c76-4dc2-a5d7-91965c8a96c2/volumes" Jul 17 18:51:37.718215 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:37.718185 2572 scope.go:117] "RemoveContainer" containerID="81e5f0325bef1ae80099baafad2830e9b9f222a22f184e3fbc7b1f1adb102263" Jul 17 18:51:37.718614 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:51:37.718374 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-f44tz_odh-ai-gateway-infra(f547776d-5ed6-476a-a1c0-91c47dad660f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz" podUID="f547776d-5ed6-476a-a1c0-91c47dad660f" Jul 17 18:51:45.647850 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:45.647821 2572 scope.go:117] "RemoveContainer" containerID="e81e48b1d4be65cae68c51d11bca22cc1d78a84f4f3b31e275df8c6389226eb6" Jul 17 18:51:50.219581 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:50.219551 2572 scope.go:117] "RemoveContainer" containerID="74f19e8fdfd02e193ec8e20e49be7bc07913175f2152abf2ff8a450ede2f72a5" Jul 17 18:51:50.778666 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:50.778633 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz" event={"ID":"a93079ad-108a-4dec-89cb-c4c3887460c5","Type":"ContainerStarted","Data":"78aa300e45052a85768dd08c954ea7340e854992a2bdf97a9762bad32410ef94"} Jul 17 18:51:50.793993 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:50.793870 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz" podStartSLOduration=19.793850887 podStartE2EDuration="19.793850887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:51:50.79273241 +0000 UTC m=+3306.202160016" watchObservedRunningTime="2026-07-17 18:51:50.793850887 +0000 UTC m=+3306.203278493" Jul 17 18:51:51.219559 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:51.219479 2572 scope.go:117] "RemoveContainer" containerID="81e5f0325bef1ae80099baafad2830e9b9f222a22f184e3fbc7b1f1adb102263" Jul 17 18:51:51.784584 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:51.784545 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz" event={"ID":"f547776d-5ed6-476a-a1c0-91c47dad660f","Type":"ContainerStarted","Data":"f920bb31b8177f5f2a869bdc9a36c4daf1eee70eed40a887ae9e2ddc10400b47"} Jul 17 18:51:51.786265 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:51.786240 2572 generic.go:356] "Generic (PLEG): container finished" podID="a93079ad-108a-4dec-89cb-c4c3887460c5" containerID="78aa300e45052a85768dd08c954ea7340e854992a2bdf97a9762bad32410ef94" exitCode=6 Jul 17 18:51:51.786377 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:51.786268 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz" event={"ID":"a93079ad-108a-4dec-89cb-c4c3887460c5","Type":"ContainerDied","Data":"78aa300e45052a85768dd08c954ea7340e854992a2bdf97a9762bad32410ef94"} Jul 17 18:51:51.786377 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:51.786312 2572 scope.go:117] "RemoveContainer" containerID="74f19e8fdfd02e193ec8e20e49be7bc07913175f2152abf2ff8a450ede2f72a5" Jul 17 18:51:51.786538 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:51.786523 2572 scope.go:117] "RemoveContainer" containerID="78aa300e45052a85768dd08c954ea7340e854992a2bdf97a9762bad32410ef94" Jul 17 18:51:51.786711 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:51:51.786694 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-747tz_odh-ai-gateway-infra(a93079ad-108a-4dec-89cb-c4c3887460c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz" podUID="a93079ad-108a-4dec-89cb-c4c3887460c5" Jul 17 18:51:51.804167 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:51.804141 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz"] Jul 17 18:51:52.793975 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:52.793943 2572 generic.go:356] "Generic (PLEG): container finished" podID="f547776d-5ed6-476a-a1c0-91c47dad660f" containerID="f920bb31b8177f5f2a869bdc9a36c4daf1eee70eed40a887ae9e2ddc10400b47" exitCode=6 Jul 17 18:51:52.794488 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:52.793990 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz" event={"ID":"f547776d-5ed6-476a-a1c0-91c47dad660f","Type":"ContainerDied","Data":"f920bb31b8177f5f2a869bdc9a36c4daf1eee70eed40a887ae9e2ddc10400b47"} Jul 17 18:51:52.794488 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:52.794053 2572 scope.go:117] "RemoveContainer" containerID="81e5f0325bef1ae80099baafad2830e9b9f222a22f184e3fbc7b1f1adb102263" Jul 17 18:51:52.794488 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:52.794300 2572 scope.go:117] "RemoveContainer" containerID="f920bb31b8177f5f2a869bdc9a36c4daf1eee70eed40a887ae9e2ddc10400b47" Jul 17 18:51:52.794488 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:51:52.794481 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-f44tz_odh-ai-gateway-infra(f547776d-5ed6-476a-a1c0-91c47dad660f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz" podUID="f547776d-5ed6-476a-a1c0-91c47dad660f" Jul 17 18:51:52.811755 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:52.811731 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz"] Jul 17 18:51:52.922885 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:52.922864 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz" Jul 17 18:51:52.982083 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:52.982054 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a93079ad-108a-4dec-89cb-c4c3887460c5-maas-api-service-ca\") pod \"a93079ad-108a-4dec-89cb-c4c3887460c5\" (UID: \"a93079ad-108a-4dec-89cb-c4c3887460c5\") " Jul 17 18:51:52.982393 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:52.982370 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93079ad-108a-4dec-89cb-c4c3887460c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a93079ad-108a-4dec-89cb-c4c3887460c5" (UID: "a93079ad-108a-4dec-89cb-c4c3887460c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:51:53.083253 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:53.083161 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a93079ad-108a-4dec-89cb-c4c3887460c5-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:51:53.801447 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:53.801421 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz" Jul 17 18:51:53.801813 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:53.801420 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz" event={"ID":"a93079ad-108a-4dec-89cb-c4c3887460c5","Type":"ContainerDied","Data":"be4aabe8d4d14dbb8af80052e59befac806554f7b6b799bf4de70ec4a3fe162b"} Jul 17 18:51:53.801813 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:53.801591 2572 scope.go:117] "RemoveContainer" containerID="78aa300e45052a85768dd08c954ea7340e854992a2bdf97a9762bad32410ef94" Jul 17 18:51:53.833316 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:53.833282 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz"] Jul 17 18:51:53.834846 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:53.834826 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-747tz"] Jul 17 18:51:53.930868 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:53.930844 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz" Jul 17 18:51:53.990845 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:53.990805 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f547776d-5ed6-476a-a1c0-91c47dad660f-maas-api-service-ca\") pod \"f547776d-5ed6-476a-a1c0-91c47dad660f\" (UID: \"f547776d-5ed6-476a-a1c0-91c47dad660f\") " Jul 17 18:51:53.991219 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:53.991196 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f547776d-5ed6-476a-a1c0-91c47dad660f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f547776d-5ed6-476a-a1c0-91c47dad660f" (UID: "f547776d-5ed6-476a-a1c0-91c47dad660f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:51:54.091940 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:54.091845 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f547776d-5ed6-476a-a1c0-91c47dad660f-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:51:54.807359 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:54.807331 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz" Jul 17 18:51:54.807853 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:54.807364 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz" event={"ID":"f547776d-5ed6-476a-a1c0-91c47dad660f","Type":"ContainerDied","Data":"26657b80932a42ba214a71427867b42281cac180333a7d0670ca5f8197f7264a"} Jul 17 18:51:54.807853 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:54.807425 2572 scope.go:117] "RemoveContainer" containerID="f920bb31b8177f5f2a869bdc9a36c4daf1eee70eed40a887ae9e2ddc10400b47" Jul 17 18:51:54.830369 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:54.830342 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz"] Jul 17 18:51:54.836222 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:54.836198 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-f44tz"] Jul 17 18:51:55.224326 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:55.224246 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a93079ad-108a-4dec-89cb-c4c3887460c5" path="/var/lib/kubelet/pods/a93079ad-108a-4dec-89cb-c4c3887460c5/volumes" Jul 17 18:51:55.224585 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:51:55.224573 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f547776d-5ed6-476a-a1c0-91c47dad660f" path="/var/lib/kubelet/pods/f547776d-5ed6-476a-a1c0-91c47dad660f/volumes" Jul 17 18:52:04.600138 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600105 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs"] Jul 17 18:52:04.600517 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600406 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93079ad-108a-4dec-89cb-c4c3887460c5" containerName="revoke-keys" Jul 17 18:52:04.600517 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600417 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93079ad-108a-4dec-89cb-c4c3887460c5" containerName="revoke-keys" Jul 17 18:52:04.600517 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600424 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f547776d-5ed6-476a-a1c0-91c47dad660f" containerName="revoke-keys" Jul 17 18:52:04.600517 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600430 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f547776d-5ed6-476a-a1c0-91c47dad660f" containerName="revoke-keys" Jul 17 18:52:04.600517 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600437 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e94ca7-3c76-4dc2-a5d7-91965c8a96c2" containerName="revoke-keys" Jul 17 18:52:04.600517 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600442 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e94ca7-3c76-4dc2-a5d7-91965c8a96c2" containerName="revoke-keys" Jul 17 18:52:04.600517 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600448 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e94ca7-3c76-4dc2-a5d7-91965c8a96c2" containerName="revoke-keys" Jul 17 18:52:04.600517 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600453 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e94ca7-3c76-4dc2-a5d7-91965c8a96c2" containerName="revoke-keys" Jul 17 18:52:04.600517 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600469 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f547776d-5ed6-476a-a1c0-91c47dad660f" containerName="revoke-keys" Jul 17 18:52:04.600517 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600474 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f547776d-5ed6-476a-a1c0-91c47dad660f" containerName="revoke-keys" Jul 17 18:52:04.600517 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600480 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e94ca7-3c76-4dc2-a5d7-91965c8a96c2" containerName="revoke-keys" Jul 17 18:52:04.600517 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600485 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e94ca7-3c76-4dc2-a5d7-91965c8a96c2" containerName="revoke-keys" Jul 17 18:52:04.600517 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600492 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93079ad-108a-4dec-89cb-c4c3887460c5" containerName="revoke-keys" Jul 17 18:52:04.600517 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600498 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93079ad-108a-4dec-89cb-c4c3887460c5" containerName="revoke-keys" Jul 17 18:52:04.600963 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600544 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f547776d-5ed6-476a-a1c0-91c47dad660f" containerName="revoke-keys" Jul 17 18:52:04.600963 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600551 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a93079ad-108a-4dec-89cb-c4c3887460c5" containerName="revoke-keys" Jul 17 18:52:04.600963 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600558 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a93079ad-108a-4dec-89cb-c4c3887460c5" containerName="revoke-keys" Jul 17 18:52:04.600963 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600564 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e94ca7-3c76-4dc2-a5d7-91965c8a96c2" containerName="revoke-keys" Jul 17 18:52:04.600963 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600569 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f547776d-5ed6-476a-a1c0-91c47dad660f" containerName="revoke-keys" Jul 17 18:52:04.600963 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600575 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e94ca7-3c76-4dc2-a5d7-91965c8a96c2" containerName="revoke-keys" Jul 17 18:52:04.600963 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600581 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e94ca7-3c76-4dc2-a5d7-91965c8a96c2" containerName="revoke-keys" Jul 17 18:52:04.600963 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.600587 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f547776d-5ed6-476a-a1c0-91c47dad660f" containerName="revoke-keys" Jul 17 18:52:04.603446 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.603431 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.606398 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.606368 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-52fc47\"/\"kube-root-ca.crt\"" Jul 17 18:52:04.607337 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.607320 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-52fc47\"/\"default-dockercfg-b9hbn\"" Jul 17 18:52:04.607413 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.607392 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-52fc47\"/\"openshift-service-ca.crt\"" Jul 17 18:52:04.607742 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.607720 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-52fc47\"/\"auth-test-model-52fc47-kserve-self-signed-certs\"" Jul 17 18:52:04.612352 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.612330 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs"] Jul 17 18:52:04.785003 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.784971 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-tmp-dir\") pod \"auth-test-model-52fc47-kserve-d589fff6c-6sbhs\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.785234 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.785058 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-dshm\") pod \"auth-test-model-52fc47-kserve-d589fff6c-6sbhs\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.785234 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.785087 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-home\") pod \"auth-test-model-52fc47-kserve-d589fff6c-6sbhs\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.785234 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.785175 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsz25\" (UniqueName: \"kubernetes.io/projected/851e200f-6bbb-48ae-8f88-dd942f80f5c9-kube-api-access-gsz25\") pod \"auth-test-model-52fc47-kserve-d589fff6c-6sbhs\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.785234 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.785211 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851e200f-6bbb-48ae-8f88-dd942f80f5c9-tls-certs\") pod \"auth-test-model-52fc47-kserve-d589fff6c-6sbhs\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.785410 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.785235 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-model-cache\") pod \"auth-test-model-52fc47-kserve-d589fff6c-6sbhs\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.886473 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.886392 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsz25\" (UniqueName: \"kubernetes.io/projected/851e200f-6bbb-48ae-8f88-dd942f80f5c9-kube-api-access-gsz25\") pod \"auth-test-model-52fc47-kserve-d589fff6c-6sbhs\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.886473 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.886434 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851e200f-6bbb-48ae-8f88-dd942f80f5c9-tls-certs\") pod \"auth-test-model-52fc47-kserve-d589fff6c-6sbhs\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.886473 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.886461 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-model-cache\") pod \"auth-test-model-52fc47-kserve-d589fff6c-6sbhs\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.886746 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.886505 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-tmp-dir\") pod \"auth-test-model-52fc47-kserve-d589fff6c-6sbhs\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.886746 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.886553 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-dshm\") pod \"auth-test-model-52fc47-kserve-d589fff6c-6sbhs\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.886746 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.886579 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-home\") pod \"auth-test-model-52fc47-kserve-d589fff6c-6sbhs\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.886934 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.886911 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-model-cache\") pod \"auth-test-model-52fc47-kserve-d589fff6c-6sbhs\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.886996 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.886936 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-tmp-dir\") pod \"auth-test-model-52fc47-kserve-d589fff6c-6sbhs\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.886996 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.886984 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-home\") pod \"auth-test-model-52fc47-kserve-d589fff6c-6sbhs\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.888822 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.888800 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-dshm\") pod \"auth-test-model-52fc47-kserve-d589fff6c-6sbhs\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.888888 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.888842 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851e200f-6bbb-48ae-8f88-dd942f80f5c9-tls-certs\") pod \"auth-test-model-52fc47-kserve-d589fff6c-6sbhs\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.894217 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.894196 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsz25\" (UniqueName: \"kubernetes.io/projected/851e200f-6bbb-48ae-8f88-dd942f80f5c9-kube-api-access-gsz25\") pod \"auth-test-model-52fc47-kserve-d589fff6c-6sbhs\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:04.913213 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:04.913190 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:05.038484 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:05.038451 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs"] Jul 17 18:52:05.041701 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:52:05.041669 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod851e200f_6bbb_48ae_8f88_dd942f80f5c9.slice/crio-43cccd11c4aab78d31e3dbdb0834f85754da62afcb190142ac7d88e0ed2b47b5 WatchSource:0}: Error finding container 43cccd11c4aab78d31e3dbdb0834f85754da62afcb190142ac7d88e0ed2b47b5: Status 404 returned error can't find the container with id 43cccd11c4aab78d31e3dbdb0834f85754da62afcb190142ac7d88e0ed2b47b5 Jul 17 18:52:05.858533 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:05.858456 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" event={"ID":"851e200f-6bbb-48ae-8f88-dd942f80f5c9","Type":"ContainerStarted","Data":"735b66fcb6cef492e8fc75877a38b2acfff321491e7120349588775d504823d8"} Jul 17 18:52:05.858533 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:05.858503 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" event={"ID":"851e200f-6bbb-48ae-8f88-dd942f80f5c9","Type":"ContainerStarted","Data":"43cccd11c4aab78d31e3dbdb0834f85754da62afcb190142ac7d88e0ed2b47b5"} Jul 17 18:52:05.875395 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:05.875345 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" podStartSLOduration=1.8753305070000001 podStartE2EDuration="1.875330507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:52:05.874087527 +0000 UTC m=+3321.283515134" watchObservedRunningTime="2026-07-17 18:52:05.875330507 +0000 UTC m=+3321.284758144" Jul 17 18:52:14.913934 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:14.913900 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:14.914344 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:14.913955 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:14.926147 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:14.926124 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:15.786963 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:15.786928 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7"] Jul 17 18:52:15.787265 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:15.787252 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93079ad-108a-4dec-89cb-c4c3887460c5" containerName="revoke-keys" Jul 17 18:52:15.787312 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:15.787267 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93079ad-108a-4dec-89cb-c4c3887460c5" containerName="revoke-keys" Jul 17 18:52:15.787312 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:15.787285 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f547776d-5ed6-476a-a1c0-91c47dad660f" containerName="revoke-keys" Jul 17 18:52:15.787312 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:15.787290 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f547776d-5ed6-476a-a1c0-91c47dad660f" containerName="revoke-keys" Jul 17 18:52:15.787411 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:15.787343 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a93079ad-108a-4dec-89cb-c4c3887460c5" containerName="revoke-keys" Jul 17 18:52:15.790401 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:15.790386 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7" Jul 17 18:52:15.792879 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:15.792858 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:52:15.793089 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:15.792899 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:52:15.796882 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:15.796862 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7"] Jul 17 18:52:15.882029 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:15.881953 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7\" (UID: \"8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7" Jul 17 18:52:15.914438 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:15.914404 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:52:15.983335 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:15.983301 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7\" (UID: \"8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7" Jul 17 18:52:15.983947 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:15.983929 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7\" (UID: \"8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7" Jul 17 18:52:16.101787 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:16.101708 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7" Jul 17 18:52:16.228092 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:16.228066 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7"] Jul 17 18:52:16.229844 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:52:16.229818 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c49ec6a_1f25_4c30_a5fc_de63f5e16e1d.slice/crio-95c81d3c81d5a8f6938d678193969ba82c0b75861c00bf04d2dc45bb3ddcf344 WatchSource:0}: Error finding container 95c81d3c81d5a8f6938d678193969ba82c0b75861c00bf04d2dc45bb3ddcf344: Status 404 returned error can't find the container with id 95c81d3c81d5a8f6938d678193969ba82c0b75861c00bf04d2dc45bb3ddcf344 Jul 17 18:52:16.908542 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:16.908508 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7" event={"ID":"8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d","Type":"ContainerStarted","Data":"f735df3ea6e5c1ccec78c3ce2d2bb05c8e76b1ddfbae25fe31af41cd3036d17a"} Jul 17 18:52:16.908713 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:16.908561 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7" event={"ID":"8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d","Type":"ContainerStarted","Data":"95c81d3c81d5a8f6938d678193969ba82c0b75861c00bf04d2dc45bb3ddcf344"} Jul 17 18:52:16.924952 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:16.924903 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7" podStartSLOduration=1.924888844 podStartE2EDuration="1.924888844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:52:16.923533571 +0000 UTC m=+3332.332961201" watchObservedRunningTime="2026-07-17 18:52:16.924888844 +0000 UTC m=+3332.334316452" Jul 17 18:52:17.914766 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:17.914729 2572 generic.go:356] "Generic (PLEG): container finished" podID="8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d" containerID="f735df3ea6e5c1ccec78c3ce2d2bb05c8e76b1ddfbae25fe31af41cd3036d17a" exitCode=6 Jul 17 18:52:17.914986 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:17.914962 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7" event={"ID":"8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d","Type":"ContainerDied","Data":"f735df3ea6e5c1ccec78c3ce2d2bb05c8e76b1ddfbae25fe31af41cd3036d17a"} Jul 17 18:52:17.915224 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:17.915208 2572 scope.go:117] "RemoveContainer" containerID="f735df3ea6e5c1ccec78c3ce2d2bb05c8e76b1ddfbae25fe31af41cd3036d17a" Jul 17 18:52:17.916882 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:17.916859 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq"] Jul 17 18:52:17.920166 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:17.920132 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq" Jul 17 18:52:17.928511 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:17.928355 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq"] Jul 17 18:52:18.001282 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:18.001252 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d3fa246-58ec-40dc-ad5b-4ad873e7e028-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq\" (UID: \"1d3fa246-58ec-40dc-ad5b-4ad873e7e028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq" Jul 17 18:52:18.102290 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:18.102260 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d3fa246-58ec-40dc-ad5b-4ad873e7e028-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq\" (UID: \"1d3fa246-58ec-40dc-ad5b-4ad873e7e028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq" Jul 17 18:52:18.102808 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:18.102790 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d3fa246-58ec-40dc-ad5b-4ad873e7e028-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq\" (UID: \"1d3fa246-58ec-40dc-ad5b-4ad873e7e028\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq" Jul 17 18:52:18.251196 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:18.251112 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq" Jul 17 18:52:18.400261 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:18.400238 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq"] Jul 17 18:52:18.402269 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:52:18.402240 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3fa246_58ec_40dc_ad5b_4ad873e7e028.slice/crio-453f286be187210163efb0467f8b27ad75503b471caefbc1dd8d00f53b7744e9 WatchSource:0}: Error finding container 453f286be187210163efb0467f8b27ad75503b471caefbc1dd8d00f53b7744e9: Status 404 returned error can't find the container with id 453f286be187210163efb0467f8b27ad75503b471caefbc1dd8d00f53b7744e9 Jul 17 18:52:18.921824 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:18.921727 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq" event={"ID":"1d3fa246-58ec-40dc-ad5b-4ad873e7e028","Type":"ContainerStarted","Data":"a78a6d686f1ce3426dfb68d3aa2c8ace4f18ceee290325b7545a82a1315cc6c6"} Jul 17 18:52:18.921824 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:18.921789 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq" event={"ID":"1d3fa246-58ec-40dc-ad5b-4ad873e7e028","Type":"ContainerStarted","Data":"453f286be187210163efb0467f8b27ad75503b471caefbc1dd8d00f53b7744e9"} Jul 17 18:52:18.923525 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:18.923498 2572 generic.go:356] "Generic (PLEG): container finished" podID="8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d" containerID="1c01d594e0b159535b980ac88504c4a39a5f9251c43d8c6976ff55241606893e" exitCode=6 Jul 17 18:52:18.923657 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:18.923584 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7" event={"ID":"8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d","Type":"ContainerDied","Data":"1c01d594e0b159535b980ac88504c4a39a5f9251c43d8c6976ff55241606893e"} Jul 17 18:52:18.923657 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:18.923635 2572 scope.go:117] "RemoveContainer" containerID="f735df3ea6e5c1ccec78c3ce2d2bb05c8e76b1ddfbae25fe31af41cd3036d17a" Jul 17 18:52:18.923747 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:18.923731 2572 scope.go:117] "RemoveContainer" containerID="1c01d594e0b159535b980ac88504c4a39a5f9251c43d8c6976ff55241606893e" Jul 17 18:52:18.923939 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:52:18.923916 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7_odh-ai-gateway-infra(8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7" podUID="8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d" Jul 17 18:52:18.937880 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:18.937842 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq" podStartSLOduration=1.937828509 podStartE2EDuration="1.937828509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:52:18.935692059 +0000 UTC m=+3334.345119688" watchObservedRunningTime="2026-07-17 18:52:18.937828509 +0000 UTC m=+3334.347256114" Jul 17 18:52:19.929757 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:19.929721 2572 generic.go:356] "Generic (PLEG): container finished" podID="1d3fa246-58ec-40dc-ad5b-4ad873e7e028" containerID="a78a6d686f1ce3426dfb68d3aa2c8ace4f18ceee290325b7545a82a1315cc6c6" exitCode=6 Jul 17 18:52:19.929951 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:19.929814 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq" event={"ID":"1d3fa246-58ec-40dc-ad5b-4ad873e7e028","Type":"ContainerDied","Data":"a78a6d686f1ce3426dfb68d3aa2c8ace4f18ceee290325b7545a82a1315cc6c6"} Jul 17 18:52:19.930029 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:19.929967 2572 scope.go:117] "RemoveContainer" containerID="a78a6d686f1ce3426dfb68d3aa2c8ace4f18ceee290325b7545a82a1315cc6c6" Jul 17 18:52:19.931631 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:19.931614 2572 scope.go:117] "RemoveContainer" containerID="1c01d594e0b159535b980ac88504c4a39a5f9251c43d8c6976ff55241606893e" Jul 17 18:52:19.931792 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:52:19.931775 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7_odh-ai-gateway-infra(8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7" podUID="8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d" Jul 17 18:52:20.570266 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:52:20.570238 2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3fa246_58ec_40dc_ad5b_4ad873e7e028.slice/crio-conmon-9fa95aa2bc7775ffdcd444933af06ca118aed203556670724dd8141f72bfa03d.scope\": RecentStats: unable to find data in memory cache]" Jul 17 18:52:20.938326 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:20.938231 2572 generic.go:356] "Generic (PLEG): container finished" podID="1d3fa246-58ec-40dc-ad5b-4ad873e7e028" containerID="9fa95aa2bc7775ffdcd444933af06ca118aed203556670724dd8141f72bfa03d" exitCode=6 Jul 17 18:52:20.938488 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:20.938313 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq" event={"ID":"1d3fa246-58ec-40dc-ad5b-4ad873e7e028","Type":"ContainerDied","Data":"9fa95aa2bc7775ffdcd444933af06ca118aed203556670724dd8141f72bfa03d"} Jul 17 18:52:20.938488 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:20.938409 2572 scope.go:117] "RemoveContainer" containerID="a78a6d686f1ce3426dfb68d3aa2c8ace4f18ceee290325b7545a82a1315cc6c6" Jul 17 18:52:20.938488 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:20.938476 2572 scope.go:117] "RemoveContainer" containerID="9fa95aa2bc7775ffdcd444933af06ca118aed203556670724dd8141f72bfa03d" Jul 17 18:52:20.938736 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:52:20.938711 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq_odh-ai-gateway-infra(1d3fa246-58ec-40dc-ad5b-4ad873e7e028)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq" podUID="1d3fa246-58ec-40dc-ad5b-4ad873e7e028" Jul 17 18:52:21.944425 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:21.944395 2572 scope.go:117] "RemoveContainer" containerID="9fa95aa2bc7775ffdcd444933af06ca118aed203556670724dd8141f72bfa03d" Jul 17 18:52:21.944929 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:52:21.944576 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq_odh-ai-gateway-infra(1d3fa246-58ec-40dc-ad5b-4ad873e7e028)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq" podUID="1d3fa246-58ec-40dc-ad5b-4ad873e7e028" Jul 17 18:52:31.402190 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:31.402158 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs"] Jul 17 18:52:31.405409 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:31.405393 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs" Jul 17 18:52:31.412024 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:31.411980 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs"] Jul 17 18:52:31.524928 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:31.524888 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/506aa908-961d-4a88-b55d-6475dbc24929-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs\" (UID: \"506aa908-961d-4a88-b55d-6475dbc24929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs" Jul 17 18:52:31.625452 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:31.625420 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/506aa908-961d-4a88-b55d-6475dbc24929-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs\" (UID: \"506aa908-961d-4a88-b55d-6475dbc24929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs" Jul 17 18:52:31.625999 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:31.625981 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/506aa908-961d-4a88-b55d-6475dbc24929-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs\" (UID: \"506aa908-961d-4a88-b55d-6475dbc24929\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs" Jul 17 18:52:31.716965 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:31.716883 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs" Jul 17 18:52:31.840777 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:31.840717 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs"] Jul 17 18:52:31.842926 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:52:31.842883 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506aa908_961d_4a88_b55d_6475dbc24929.slice/crio-7a4d9fded9cabdd2c44d48f54856ee37311146ac25205f4a76ab3cd2d5dfa5ae WatchSource:0}: Error finding container 7a4d9fded9cabdd2c44d48f54856ee37311146ac25205f4a76ab3cd2d5dfa5ae: Status 404 returned error can't find the container with id 7a4d9fded9cabdd2c44d48f54856ee37311146ac25205f4a76ab3cd2d5dfa5ae Jul 17 18:52:31.988185 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:31.988101 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs" event={"ID":"506aa908-961d-4a88-b55d-6475dbc24929","Type":"ContainerStarted","Data":"7a4d9fded9cabdd2c44d48f54856ee37311146ac25205f4a76ab3cd2d5dfa5ae"} Jul 17 18:52:32.994429 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:32.994397 2572 generic.go:356] "Generic (PLEG): container finished" podID="506aa908-961d-4a88-b55d-6475dbc24929" containerID="f9b892467ff81415938d672ef0163f4d0cb44c616148dc2f19cccda9f4337f11" exitCode=6 Jul 17 18:52:32.994912 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:32.994479 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs" event={"ID":"506aa908-961d-4a88-b55d-6475dbc24929","Type":"ContainerDied","Data":"f9b892467ff81415938d672ef0163f4d0cb44c616148dc2f19cccda9f4337f11"} Jul 17 18:52:32.994912 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:32.994693 2572 scope.go:117] "RemoveContainer" containerID="f9b892467ff81415938d672ef0163f4d0cb44c616148dc2f19cccda9f4337f11" Jul 17 18:52:33.226361 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:33.219889 2572 scope.go:117] "RemoveContainer" containerID="1c01d594e0b159535b980ac88504c4a39a5f9251c43d8c6976ff55241606893e" Jul 17 18:52:33.966344 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:33.966268 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67"] Jul 17 18:52:33.969498 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:33.969482 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67" Jul 17 18:52:33.978867 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:33.978839 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67"] Jul 17 18:52:34.000764 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:34.000739 2572 generic.go:356] "Generic (PLEG): container finished" podID="506aa908-961d-4a88-b55d-6475dbc24929" containerID="1a354c871ccc029ae008c2c470d663ba370699cde7cfeca08f23998a965eb10d" exitCode=6 Jul 17 18:52:34.001187 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:34.000831 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs" event={"ID":"506aa908-961d-4a88-b55d-6475dbc24929","Type":"ContainerDied","Data":"1a354c871ccc029ae008c2c470d663ba370699cde7cfeca08f23998a965eb10d"} Jul 17 18:52:34.001187 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:34.000900 2572 scope.go:117] "RemoveContainer" containerID="f9b892467ff81415938d672ef0163f4d0cb44c616148dc2f19cccda9f4337f11" Jul 17 18:52:34.001187 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:34.001068 2572 scope.go:117] "RemoveContainer" containerID="1a354c871ccc029ae008c2c470d663ba370699cde7cfeca08f23998a965eb10d" Jul 17 18:52:34.001363 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:52:34.001300 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs_odh-ai-gateway-infra(506aa908-961d-4a88-b55d-6475dbc24929)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs" podUID="506aa908-961d-4a88-b55d-6475dbc24929" Jul 17 18:52:34.002837 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:34.002815 2572 generic.go:356] "Generic (PLEG): container finished" podID="8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d" containerID="7165b07cd52a8734249f847e9cffb5e2ae0fd7db91ff3c0e3ea850c2a2be386d" exitCode=6 Jul 17 18:52:34.002928 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:34.002881 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7" event={"ID":"8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d","Type":"ContainerDied","Data":"7165b07cd52a8734249f847e9cffb5e2ae0fd7db91ff3c0e3ea850c2a2be386d"} Jul 17 18:52:34.003141 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:34.003128 2572 scope.go:117] "RemoveContainer" containerID="7165b07cd52a8734249f847e9cffb5e2ae0fd7db91ff3c0e3ea850c2a2be386d" Jul 17 18:52:34.003282 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:52:34.003266 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7_odh-ai-gateway-infra(8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7" podUID="8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d" Jul 17 18:52:34.011538 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:34.011524 2572 scope.go:117] "RemoveContainer" containerID="1c01d594e0b159535b980ac88504c4a39a5f9251c43d8c6976ff55241606893e" Jul 17 18:52:34.047658 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:34.047632 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4714b773-c651-4812-9733-8c20c1ccd5b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-g8s67\" (UID: \"4714b773-c651-4812-9733-8c20c1ccd5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67" Jul 17 18:52:34.148834 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:34.148805 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4714b773-c651-4812-9733-8c20c1ccd5b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-g8s67\" (UID: \"4714b773-c651-4812-9733-8c20c1ccd5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67" Jul 17 18:52:34.149492 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:34.149462 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4714b773-c651-4812-9733-8c20c1ccd5b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-g8s67\" (UID: \"4714b773-c651-4812-9733-8c20c1ccd5b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67" Jul 17 18:52:34.219906 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:34.219832 2572 scope.go:117] "RemoveContainer" containerID="9fa95aa2bc7775ffdcd444933af06ca118aed203556670724dd8141f72bfa03d" Jul 17 18:52:34.280863 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:34.280839 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67" Jul 17 18:52:34.409243 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:34.409216 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67"] Jul 17 18:52:34.411412 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:52:34.411380 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4714b773_c651_4812_9733_8c20c1ccd5b1.slice/crio-de596639c4af0523f5649e48c616515042d59d2e10d6620a260f5c368804cc4b WatchSource:0}: Error finding container de596639c4af0523f5649e48c616515042d59d2e10d6620a260f5c368804cc4b: Status 404 returned error can't find the container with id de596639c4af0523f5649e48c616515042d59d2e10d6620a260f5c368804cc4b Jul 17 18:52:35.010449 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:35.010411 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67" event={"ID":"4714b773-c651-4812-9733-8c20c1ccd5b1","Type":"ContainerStarted","Data":"1062967058fdef96dfd64057099594bd629e8fda112e7b6365107ee490c9728a"} Jul 17 18:52:35.010892 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:35.010467 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67" event={"ID":"4714b773-c651-4812-9733-8c20c1ccd5b1","Type":"ContainerStarted","Data":"de596639c4af0523f5649e48c616515042d59d2e10d6620a260f5c368804cc4b"} Jul 17 18:52:35.011991 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:35.011974 2572 scope.go:117] "RemoveContainer" containerID="1a354c871ccc029ae008c2c470d663ba370699cde7cfeca08f23998a965eb10d" Jul 17 18:52:35.012224 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:52:35.012199 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs_odh-ai-gateway-infra(506aa908-961d-4a88-b55d-6475dbc24929)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs" podUID="506aa908-961d-4a88-b55d-6475dbc24929" Jul 17 18:52:35.013593 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:35.013574 2572 generic.go:356] "Generic (PLEG): container finished" podID="1d3fa246-58ec-40dc-ad5b-4ad873e7e028" containerID="9d04d6f5c1f64ee1de186f02b840438ee937903530f5079f8a11cb3938b8e137" exitCode=6 Jul 17 18:52:35.013685 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:35.013600 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq" event={"ID":"1d3fa246-58ec-40dc-ad5b-4ad873e7e028","Type":"ContainerDied","Data":"9d04d6f5c1f64ee1de186f02b840438ee937903530f5079f8a11cb3938b8e137"} Jul 17 18:52:35.013685 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:35.013634 2572 scope.go:117] "RemoveContainer" containerID="9fa95aa2bc7775ffdcd444933af06ca118aed203556670724dd8141f72bfa03d" Jul 17 18:52:35.013919 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:35.013901 2572 scope.go:117] "RemoveContainer" containerID="9d04d6f5c1f64ee1de186f02b840438ee937903530f5079f8a11cb3938b8e137" Jul 17 18:52:35.014105 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:52:35.014086 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq_odh-ai-gateway-infra(1d3fa246-58ec-40dc-ad5b-4ad873e7e028)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq" podUID="1d3fa246-58ec-40dc-ad5b-4ad873e7e028" Jul 17 18:52:35.025155 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:35.025120 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67" podStartSLOduration=2.025108488 podStartE2EDuration="2.025108488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:52:35.02411684 +0000 UTC m=+3350.433544446" watchObservedRunningTime="2026-07-17 18:52:35.025108488 +0000 UTC m=+3350.434536100" Jul 17 18:52:35.037758 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:35.037739 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7"] Jul 17 18:52:35.211233 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:35.211179 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7" Jul 17 18:52:35.360491 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:35.360457 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d-maas-api-service-ca\") pod \"8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d\" (UID: \"8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d\") " Jul 17 18:52:35.360797 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:35.360775 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d" (UID: "8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:52:35.461395 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:35.461325 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:52:36.019718 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:36.019680 2572 generic.go:356] "Generic (PLEG): container finished" podID="4714b773-c651-4812-9733-8c20c1ccd5b1" containerID="1062967058fdef96dfd64057099594bd629e8fda112e7b6365107ee490c9728a" exitCode=6 Jul 17 18:52:36.020147 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:36.019732 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67" event={"ID":"4714b773-c651-4812-9733-8c20c1ccd5b1","Type":"ContainerDied","Data":"1062967058fdef96dfd64057099594bd629e8fda112e7b6365107ee490c9728a"} Jul 17 18:52:36.020147 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:36.019969 2572 scope.go:117] "RemoveContainer" containerID="1062967058fdef96dfd64057099594bd629e8fda112e7b6365107ee490c9728a" Jul 17 18:52:36.022616 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:36.022595 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7" event={"ID":"8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d","Type":"ContainerDied","Data":"95c81d3c81d5a8f6938d678193969ba82c0b75861c00bf04d2dc45bb3ddcf344"} Jul 17 18:52:36.022616 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:36.022610 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7" Jul 17 18:52:36.022785 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:36.022644 2572 scope.go:117] "RemoveContainer" containerID="7165b07cd52a8734249f847e9cffb5e2ae0fd7db91ff3c0e3ea850c2a2be386d" Jul 17 18:52:36.048897 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:36.048872 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7"] Jul 17 18:52:36.054665 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:36.054645 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-nqlx7"] Jul 17 18:52:36.063350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:36.063330 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq"] Jul 17 18:52:36.198165 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:36.198139 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq" Jul 17 18:52:36.267674 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:36.267652 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d3fa246-58ec-40dc-ad5b-4ad873e7e028-maas-api-service-ca\") pod \"1d3fa246-58ec-40dc-ad5b-4ad873e7e028\" (UID: \"1d3fa246-58ec-40dc-ad5b-4ad873e7e028\") " Jul 17 18:52:36.267986 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:36.267963 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d3fa246-58ec-40dc-ad5b-4ad873e7e028-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1d3fa246-58ec-40dc-ad5b-4ad873e7e028" (UID: "1d3fa246-58ec-40dc-ad5b-4ad873e7e028"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:52:36.368585 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:36.368559 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d3fa246-58ec-40dc-ad5b-4ad873e7e028-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:52:37.029069 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:37.029035 2572 generic.go:356] "Generic (PLEG): container finished" podID="4714b773-c651-4812-9733-8c20c1ccd5b1" containerID="2ecc16794a2d656f6e0e1a200a1871b53fe535c63b0efc6804336f0eaf3f26ee" exitCode=6 Jul 17 18:52:37.029550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:37.029106 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67" event={"ID":"4714b773-c651-4812-9733-8c20c1ccd5b1","Type":"ContainerDied","Data":"2ecc16794a2d656f6e0e1a200a1871b53fe535c63b0efc6804336f0eaf3f26ee"} Jul 17 18:52:37.029550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:37.029210 2572 scope.go:117] "RemoveContainer" containerID="1062967058fdef96dfd64057099594bd629e8fda112e7b6365107ee490c9728a" Jul 17 18:52:37.029550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:37.029363 2572 scope.go:117] "RemoveContainer" containerID="2ecc16794a2d656f6e0e1a200a1871b53fe535c63b0efc6804336f0eaf3f26ee" Jul 17 18:52:37.029734 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:52:37.029588 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-g8s67_odh-ai-gateway-infra(4714b773-c651-4812-9733-8c20c1ccd5b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67" podUID="4714b773-c651-4812-9733-8c20c1ccd5b1" Jul 17 18:52:37.030433 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:37.030418 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq" Jul 17 18:52:37.030532 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:37.030420 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq" event={"ID":"1d3fa246-58ec-40dc-ad5b-4ad873e7e028","Type":"ContainerDied","Data":"453f286be187210163efb0467f8b27ad75503b471caefbc1dd8d00f53b7744e9"} Jul 17 18:52:37.040240 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:37.040218 2572 scope.go:117] "RemoveContainer" containerID="9d04d6f5c1f64ee1de186f02b840438ee937903530f5079f8a11cb3938b8e137" Jul 17 18:52:37.055977 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:37.055955 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq"] Jul 17 18:52:37.060321 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:37.060301 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dxzvq"] Jul 17 18:52:37.224312 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:37.224282 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d3fa246-58ec-40dc-ad5b-4ad873e7e028" path="/var/lib/kubelet/pods/1d3fa246-58ec-40dc-ad5b-4ad873e7e028/volumes" Jul 17 18:52:37.224673 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:37.224658 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d" path="/var/lib/kubelet/pods/8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d/volumes" Jul 17 18:52:38.038380 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:38.038349 2572 scope.go:117] "RemoveContainer" containerID="2ecc16794a2d656f6e0e1a200a1871b53fe535c63b0efc6804336f0eaf3f26ee" Jul 17 18:52:38.038741 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:52:38.038536 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-g8s67_odh-ai-gateway-infra(4714b773-c651-4812-9733-8c20c1ccd5b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67" podUID="4714b773-c651-4812-9733-8c20c1ccd5b1" Jul 17 18:52:50.220130 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:50.220097 2572 scope.go:117] "RemoveContainer" containerID="1a354c871ccc029ae008c2c470d663ba370699cde7cfeca08f23998a965eb10d" Jul 17 18:52:51.096548 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:51.096514 2572 generic.go:356] "Generic (PLEG): container finished" podID="506aa908-961d-4a88-b55d-6475dbc24929" containerID="3d031f83e22a255d95ebd9e17a0cc675c9d168a30a4f67340a1cd8884f497051" exitCode=6 Jul 17 18:52:51.096717 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:51.096585 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs" event={"ID":"506aa908-961d-4a88-b55d-6475dbc24929","Type":"ContainerDied","Data":"3d031f83e22a255d95ebd9e17a0cc675c9d168a30a4f67340a1cd8884f497051"} Jul 17 18:52:51.096717 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:51.096646 2572 scope.go:117] "RemoveContainer" containerID="1a354c871ccc029ae008c2c470d663ba370699cde7cfeca08f23998a965eb10d" Jul 17 18:52:51.097026 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:51.096987 2572 scope.go:117] "RemoveContainer" containerID="3d031f83e22a255d95ebd9e17a0cc675c9d168a30a4f67340a1cd8884f497051" Jul 17 18:52:51.097254 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:52:51.097232 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs_odh-ai-gateway-infra(506aa908-961d-4a88-b55d-6475dbc24929)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs" podUID="506aa908-961d-4a88-b55d-6475dbc24929" Jul 17 18:52:51.219717 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:51.219630 2572 scope.go:117] "RemoveContainer" containerID="2ecc16794a2d656f6e0e1a200a1871b53fe535c63b0efc6804336f0eaf3f26ee" Jul 17 18:52:52.103002 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:52.102925 2572 generic.go:356] "Generic (PLEG): container finished" podID="4714b773-c651-4812-9733-8c20c1ccd5b1" containerID="c485929a9635a03ad1057ad304a1829de0ddcc4f289e5f5df2a48f05b86013c3" exitCode=6 Jul 17 18:52:52.103488 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:52.102999 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67" event={"ID":"4714b773-c651-4812-9733-8c20c1ccd5b1","Type":"ContainerDied","Data":"c485929a9635a03ad1057ad304a1829de0ddcc4f289e5f5df2a48f05b86013c3"} Jul 17 18:52:52.103488 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:52.103084 2572 scope.go:117] "RemoveContainer" containerID="2ecc16794a2d656f6e0e1a200a1871b53fe535c63b0efc6804336f0eaf3f26ee" Jul 17 18:52:52.103488 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:52.103406 2572 scope.go:117] "RemoveContainer" containerID="c485929a9635a03ad1057ad304a1829de0ddcc4f289e5f5df2a48f05b86013c3" Jul 17 18:52:52.103655 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:52:52.103636 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-g8s67_odh-ai-gateway-infra(4714b773-c651-4812-9733-8c20c1ccd5b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67" podUID="4714b773-c651-4812-9733-8c20c1ccd5b1" Jul 17 18:52:52.124232 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:52.124210 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs"] Jul 17 18:52:52.251137 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:52.251107 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs" Jul 17 18:52:52.401294 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:52.401218 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/506aa908-961d-4a88-b55d-6475dbc24929-maas-api-service-ca\") pod \"506aa908-961d-4a88-b55d-6475dbc24929\" (UID: \"506aa908-961d-4a88-b55d-6475dbc24929\") " Jul 17 18:52:52.401548 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:52.401526 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506aa908-961d-4a88-b55d-6475dbc24929-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "506aa908-961d-4a88-b55d-6475dbc24929" (UID: "506aa908-961d-4a88-b55d-6475dbc24929"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:52:52.502207 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:52.502178 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/506aa908-961d-4a88-b55d-6475dbc24929-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:52:53.118480 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:53.118442 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs" event={"ID":"506aa908-961d-4a88-b55d-6475dbc24929","Type":"ContainerDied","Data":"7a4d9fded9cabdd2c44d48f54856ee37311146ac25205f4a76ab3cd2d5dfa5ae"} Jul 17 18:52:53.118480 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:53.118460 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs" Jul 17 18:52:53.119174 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:53.118525 2572 scope.go:117] "RemoveContainer" containerID="3d031f83e22a255d95ebd9e17a0cc675c9d168a30a4f67340a1cd8884f497051" Jul 17 18:52:53.127855 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:53.127834 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67"] Jul 17 18:52:53.142067 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:53.142036 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs"] Jul 17 18:52:53.152146 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:53.148682 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-7plgs"] Jul 17 18:52:53.224646 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:53.224616 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="506aa908-961d-4a88-b55d-6475dbc24929" path="/var/lib/kubelet/pods/506aa908-961d-4a88-b55d-6475dbc24929/volumes" Jul 17 18:52:53.256020 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:53.255982 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67" Jul 17 18:52:53.307233 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:53.307201 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4714b773-c651-4812-9733-8c20c1ccd5b1-maas-api-service-ca\") pod \"4714b773-c651-4812-9733-8c20c1ccd5b1\" (UID: \"4714b773-c651-4812-9733-8c20c1ccd5b1\") " Jul 17 18:52:53.307553 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:53.307531 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4714b773-c651-4812-9733-8c20c1ccd5b1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4714b773-c651-4812-9733-8c20c1ccd5b1" (UID: "4714b773-c651-4812-9733-8c20c1ccd5b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:52:53.407983 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:53.407890 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4714b773-c651-4812-9733-8c20c1ccd5b1-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:52:54.124052 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:54.124024 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67" Jul 17 18:52:54.124541 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:54.124062 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67" event={"ID":"4714b773-c651-4812-9733-8c20c1ccd5b1","Type":"ContainerDied","Data":"de596639c4af0523f5649e48c616515042d59d2e10d6620a260f5c368804cc4b"} Jul 17 18:52:54.124541 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:54.124132 2572 scope.go:117] "RemoveContainer" containerID="c485929a9635a03ad1057ad304a1829de0ddcc4f289e5f5df2a48f05b86013c3" Jul 17 18:52:54.146671 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:54.146646 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67"] Jul 17 18:52:54.152949 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:54.152927 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-g8s67"] Jul 17 18:52:55.224908 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:52:55.224874 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4714b773-c651-4812-9733-8c20c1ccd5b1" path="/var/lib/kubelet/pods/4714b773-c651-4812-9733-8c20c1ccd5b1/volumes" Jul 17 18:53:07.958701 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.958665 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg"] Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.958951 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.958962 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.958970 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d3fa246-58ec-40dc-ad5b-4ad873e7e028" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.958976 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d3fa246-58ec-40dc-ad5b-4ad873e7e028" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.958982 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d3fa246-58ec-40dc-ad5b-4ad873e7e028" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.958987 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d3fa246-58ec-40dc-ad5b-4ad873e7e028" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.958995 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959000 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959023 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="506aa908-961d-4a88-b55d-6475dbc24929" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959028 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="506aa908-961d-4a88-b55d-6475dbc24929" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959039 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4714b773-c651-4812-9733-8c20c1ccd5b1" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959043 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4714b773-c651-4812-9733-8c20c1ccd5b1" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959050 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4714b773-c651-4812-9733-8c20c1ccd5b1" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959054 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4714b773-c651-4812-9733-8c20c1ccd5b1" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959061 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="506aa908-961d-4a88-b55d-6475dbc24929" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959066 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="506aa908-961d-4a88-b55d-6475dbc24929" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959071 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4714b773-c651-4812-9733-8c20c1ccd5b1" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959075 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4714b773-c651-4812-9733-8c20c1ccd5b1" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959125 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959133 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="506aa908-961d-4a88-b55d-6475dbc24929" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959138 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="506aa908-961d-4a88-b55d-6475dbc24929" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959143 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4714b773-c651-4812-9733-8c20c1ccd5b1" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959150 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d3fa246-58ec-40dc-ad5b-4ad873e7e028" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959154 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d3fa246-58ec-40dc-ad5b-4ad873e7e028" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959160 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4714b773-c651-4812-9733-8c20c1ccd5b1" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959165 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4714b773-c651-4812-9733-8c20c1ccd5b1" containerName="revoke-keys" Jul 17 18:53:07.959188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.959171 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d" containerName="revoke-keys" Jul 17 18:53:07.962146 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.962130 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg" Jul 17 18:53:07.964625 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.964608 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:53:07.964700 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.964611 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:53:07.970032 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:07.969979 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg"] Jul 17 18:53:08.027883 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:08.027849 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3317456a-b2d5-44e2-b346-8c2e42857626-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg\" (UID: \"3317456a-b2d5-44e2-b346-8c2e42857626\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg" Jul 17 18:53:08.129319 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:08.129280 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3317456a-b2d5-44e2-b346-8c2e42857626-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg\" (UID: \"3317456a-b2d5-44e2-b346-8c2e42857626\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg" Jul 17 18:53:08.129883 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:08.129866 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3317456a-b2d5-44e2-b346-8c2e42857626-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg\" (UID: \"3317456a-b2d5-44e2-b346-8c2e42857626\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg" Jul 17 18:53:08.273642 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:08.273556 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg" Jul 17 18:53:08.606084 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:08.606064 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg"] Jul 17 18:53:08.608300 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:53:08.608277 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3317456a_b2d5_44e2_b346_8c2e42857626.slice/crio-2fb3305f76964fe6e5c5a99912002a6d44f60b6c36b4cbec840e9f24ed03abd2 WatchSource:0}: Error finding container 2fb3305f76964fe6e5c5a99912002a6d44f60b6c36b4cbec840e9f24ed03abd2: Status 404 returned error can't find the container with id 2fb3305f76964fe6e5c5a99912002a6d44f60b6c36b4cbec840e9f24ed03abd2 Jul 17 18:53:09.191045 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:09.190953 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg" event={"ID":"3317456a-b2d5-44e2-b346-8c2e42857626","Type":"ContainerStarted","Data":"e4eea80ad42ad406caa06882c3854b1dfd0cc0afde914f37b870542b1667bed7"} Jul 17 18:53:09.191045 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:09.191004 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg" event={"ID":"3317456a-b2d5-44e2-b346-8c2e42857626","Type":"ContainerStarted","Data":"2fb3305f76964fe6e5c5a99912002a6d44f60b6c36b4cbec840e9f24ed03abd2"} Jul 17 18:53:09.205485 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:09.205430 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg" podStartSLOduration=2.205414086 podStartE2EDuration="2.205414086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:53:09.204649334 +0000 UTC m=+3384.614076942" watchObservedRunningTime="2026-07-17 18:53:09.205414086 +0000 UTC m=+3384.614841692" Jul 17 18:53:10.197804 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:10.197771 2572 generic.go:356] "Generic (PLEG): container finished" podID="3317456a-b2d5-44e2-b346-8c2e42857626" containerID="e4eea80ad42ad406caa06882c3854b1dfd0cc0afde914f37b870542b1667bed7" exitCode=6 Jul 17 18:53:10.198245 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:10.197859 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg" event={"ID":"3317456a-b2d5-44e2-b346-8c2e42857626","Type":"ContainerDied","Data":"e4eea80ad42ad406caa06882c3854b1dfd0cc0afde914f37b870542b1667bed7"} Jul 17 18:53:10.198245 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:10.198020 2572 scope.go:117] "RemoveContainer" containerID="e4eea80ad42ad406caa06882c3854b1dfd0cc0afde914f37b870542b1667bed7" Jul 17 18:53:11.203878 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:11.203843 2572 generic.go:356] "Generic (PLEG): container finished" podID="3317456a-b2d5-44e2-b346-8c2e42857626" containerID="ab7b1c965ce9f1683f09684715154249c4f3fba4e4c3948b4f7d8cd08767707a" exitCode=6 Jul 17 18:53:11.204314 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:11.203921 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg" event={"ID":"3317456a-b2d5-44e2-b346-8c2e42857626","Type":"ContainerDied","Data":"ab7b1c965ce9f1683f09684715154249c4f3fba4e4c3948b4f7d8cd08767707a"} Jul 17 18:53:11.204314 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:11.203963 2572 scope.go:117] "RemoveContainer" containerID="e4eea80ad42ad406caa06882c3854b1dfd0cc0afde914f37b870542b1667bed7" Jul 17 18:53:11.204314 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:11.204203 2572 scope.go:117] "RemoveContainer" containerID="ab7b1c965ce9f1683f09684715154249c4f3fba4e4c3948b4f7d8cd08767707a" Jul 17 18:53:11.204486 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:53:11.204421 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg_odh-ai-gateway-infra(3317456a-b2d5-44e2-b346-8c2e42857626)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg" podUID="3317456a-b2d5-44e2-b346-8c2e42857626" Jul 17 18:53:12.209804 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:12.209779 2572 scope.go:117] "RemoveContainer" containerID="ab7b1c965ce9f1683f09684715154249c4f3fba4e4c3948b4f7d8cd08767707a" Jul 17 18:53:12.210251 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:53:12.209997 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg_odh-ai-gateway-infra(3317456a-b2d5-44e2-b346-8c2e42857626)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg" podUID="3317456a-b2d5-44e2-b346-8c2e42857626" Jul 17 18:53:15.830649 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:15.830615 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28"] Jul 17 18:53:15.831135 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:15.830910 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d3fa246-58ec-40dc-ad5b-4ad873e7e028" containerName="revoke-keys" Jul 17 18:53:15.831135 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:15.830920 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d3fa246-58ec-40dc-ad5b-4ad873e7e028" containerName="revoke-keys" Jul 17 18:53:15.831135 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:15.830928 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d" containerName="revoke-keys" Jul 17 18:53:15.831135 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:15.830933 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d" containerName="revoke-keys" Jul 17 18:53:15.831135 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:15.830949 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="506aa908-961d-4a88-b55d-6475dbc24929" containerName="revoke-keys" Jul 17 18:53:15.831135 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:15.830955 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="506aa908-961d-4a88-b55d-6475dbc24929" containerName="revoke-keys" Jul 17 18:53:15.831135 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:15.831000 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="506aa908-961d-4a88-b55d-6475dbc24929" containerName="revoke-keys" Jul 17 18:53:15.831135 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:15.831024 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d3fa246-58ec-40dc-ad5b-4ad873e7e028" containerName="revoke-keys" Jul 17 18:53:15.831135 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:15.831032 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c49ec6a-1f25-4c30-a5fc-de63f5e16e1d" containerName="revoke-keys" Jul 17 18:53:15.835228 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:15.835212 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28" Jul 17 18:53:15.840613 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:15.840585 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28"] Jul 17 18:53:15.892536 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:15.892510 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78523e48-7ddf-40ec-ad38-e7afbeb77b55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28\" (UID: \"78523e48-7ddf-40ec-ad38-e7afbeb77b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28" Jul 17 18:53:15.993683 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:15.993650 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78523e48-7ddf-40ec-ad38-e7afbeb77b55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28\" (UID: \"78523e48-7ddf-40ec-ad38-e7afbeb77b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28" Jul 17 18:53:15.994269 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:15.994250 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78523e48-7ddf-40ec-ad38-e7afbeb77b55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28\" (UID: \"78523e48-7ddf-40ec-ad38-e7afbeb77b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28" Jul 17 18:53:16.146844 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:16.146757 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28" Jul 17 18:53:16.272095 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:16.272071 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28"] Jul 17 18:53:16.273651 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:53:16.273613 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78523e48_7ddf_40ec_ad38_e7afbeb77b55.slice/crio-f686bfcb5e9ab087fde658e70a711e5bab9aabe23a2d9666998c2681afc83ce5 WatchSource:0}: Error finding container f686bfcb5e9ab087fde658e70a711e5bab9aabe23a2d9666998c2681afc83ce5: Status 404 returned error can't find the container with id f686bfcb5e9ab087fde658e70a711e5bab9aabe23a2d9666998c2681afc83ce5 Jul 17 18:53:17.231672 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:17.231594 2572 generic.go:356] "Generic (PLEG): container finished" podID="78523e48-7ddf-40ec-ad38-e7afbeb77b55" containerID="13b5a4ac388231452cef9b95d1f98612dc62ea56a3b476bb2783fd30b62d97b8" exitCode=6 Jul 17 18:53:17.231672 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:17.231636 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28" event={"ID":"78523e48-7ddf-40ec-ad38-e7afbeb77b55","Type":"ContainerDied","Data":"13b5a4ac388231452cef9b95d1f98612dc62ea56a3b476bb2783fd30b62d97b8"} Jul 17 18:53:17.232290 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:17.231700 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28" event={"ID":"78523e48-7ddf-40ec-ad38-e7afbeb77b55","Type":"ContainerStarted","Data":"f686bfcb5e9ab087fde658e70a711e5bab9aabe23a2d9666998c2681afc83ce5"} Jul 17 18:53:17.232290 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:17.231951 2572 scope.go:117] "RemoveContainer" containerID="13b5a4ac388231452cef9b95d1f98612dc62ea56a3b476bb2783fd30b62d97b8" Jul 17 18:53:18.238046 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:18.237940 2572 generic.go:356] "Generic (PLEG): container finished" podID="78523e48-7ddf-40ec-ad38-e7afbeb77b55" containerID="68a7da39fbe4c7e3ebe578873e1c9c2e0623f00e2c7a0a58a5d49bfd437702da" exitCode=6 Jul 17 18:53:18.238046 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:18.237996 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28" event={"ID":"78523e48-7ddf-40ec-ad38-e7afbeb77b55","Type":"ContainerDied","Data":"68a7da39fbe4c7e3ebe578873e1c9c2e0623f00e2c7a0a58a5d49bfd437702da"} Jul 17 18:53:18.238567 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:18.238081 2572 scope.go:117] "RemoveContainer" containerID="13b5a4ac388231452cef9b95d1f98612dc62ea56a3b476bb2783fd30b62d97b8" Jul 17 18:53:18.238567 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:18.238303 2572 scope.go:117] "RemoveContainer" containerID="68a7da39fbe4c7e3ebe578873e1c9c2e0623f00e2c7a0a58a5d49bfd437702da" Jul 17 18:53:18.238567 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:53:18.238524 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28_odh-ai-gateway-infra(78523e48-7ddf-40ec-ad38-e7afbeb77b55)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28" podUID="78523e48-7ddf-40ec-ad38-e7afbeb77b55" Jul 17 18:53:19.244173 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:19.244145 2572 scope.go:117] "RemoveContainer" containerID="68a7da39fbe4c7e3ebe578873e1c9c2e0623f00e2c7a0a58a5d49bfd437702da" Jul 17 18:53:19.244542 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:53:19.244328 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28_odh-ai-gateway-infra(78523e48-7ddf-40ec-ad38-e7afbeb77b55)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28" podUID="78523e48-7ddf-40ec-ad38-e7afbeb77b55" Jul 17 18:53:27.219711 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:27.219675 2572 scope.go:117] "RemoveContainer" containerID="ab7b1c965ce9f1683f09684715154249c4f3fba4e4c3948b4f7d8cd08767707a" Jul 17 18:53:28.284994 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:28.284961 2572 generic.go:356] "Generic (PLEG): container finished" podID="3317456a-b2d5-44e2-b346-8c2e42857626" containerID="599c7b3fb156740019cba4fed1214c3a65c9ee00d3767d9333607fb3b7d6632f" exitCode=6 Jul 17 18:53:28.285404 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:28.284994 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg" event={"ID":"3317456a-b2d5-44e2-b346-8c2e42857626","Type":"ContainerDied","Data":"599c7b3fb156740019cba4fed1214c3a65c9ee00d3767d9333607fb3b7d6632f"} Jul 17 18:53:28.285404 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:28.285080 2572 scope.go:117] "RemoveContainer" containerID="ab7b1c965ce9f1683f09684715154249c4f3fba4e4c3948b4f7d8cd08767707a" Jul 17 18:53:28.285404 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:28.285377 2572 scope.go:117] "RemoveContainer" containerID="599c7b3fb156740019cba4fed1214c3a65c9ee00d3767d9333607fb3b7d6632f" Jul 17 18:53:28.285894 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:53:28.285603 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg_odh-ai-gateway-infra(3317456a-b2d5-44e2-b346-8c2e42857626)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg" podUID="3317456a-b2d5-44e2-b346-8c2e42857626" Jul 17 18:53:29.309636 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:29.309604 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg"] Jul 17 18:53:29.436391 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:29.436371 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg" Jul 17 18:53:29.514945 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:29.514906 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3317456a-b2d5-44e2-b346-8c2e42857626-maas-api-service-ca\") pod \"3317456a-b2d5-44e2-b346-8c2e42857626\" (UID: \"3317456a-b2d5-44e2-b346-8c2e42857626\") " Jul 17 18:53:29.515320 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:29.515302 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3317456a-b2d5-44e2-b346-8c2e42857626-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3317456a-b2d5-44e2-b346-8c2e42857626" (UID: "3317456a-b2d5-44e2-b346-8c2e42857626"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:53:29.616508 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:29.616434 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3317456a-b2d5-44e2-b346-8c2e42857626-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:53:30.295911 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:30.295880 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg" Jul 17 18:53:30.296088 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:30.295911 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg" event={"ID":"3317456a-b2d5-44e2-b346-8c2e42857626","Type":"ContainerDied","Data":"2fb3305f76964fe6e5c5a99912002a6d44f60b6c36b4cbec840e9f24ed03abd2"} Jul 17 18:53:30.296088 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:30.295970 2572 scope.go:117] "RemoveContainer" containerID="599c7b3fb156740019cba4fed1214c3a65c9ee00d3767d9333607fb3b7d6632f" Jul 17 18:53:30.320291 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:30.320265 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg"] Jul 17 18:53:30.324495 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:30.324473 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-dqqhg"] Jul 17 18:53:31.224400 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:31.224372 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3317456a-b2d5-44e2-b346-8c2e42857626" path="/var/lib/kubelet/pods/3317456a-b2d5-44e2-b346-8c2e42857626/volumes" Jul 17 18:53:31.443510 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:31.443475 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz"] Jul 17 18:53:31.443870 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:31.443765 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3317456a-b2d5-44e2-b346-8c2e42857626" containerName="revoke-keys" Jul 17 18:53:31.443870 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:31.443775 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3317456a-b2d5-44e2-b346-8c2e42857626" containerName="revoke-keys" Jul 17 18:53:31.443870 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:31.443785 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3317456a-b2d5-44e2-b346-8c2e42857626" containerName="revoke-keys" Jul 17 18:53:31.443870 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:31.443790 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3317456a-b2d5-44e2-b346-8c2e42857626" containerName="revoke-keys" Jul 17 18:53:31.443870 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:31.443801 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3317456a-b2d5-44e2-b346-8c2e42857626" containerName="revoke-keys" Jul 17 18:53:31.443870 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:31.443806 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3317456a-b2d5-44e2-b346-8c2e42857626" containerName="revoke-keys" Jul 17 18:53:31.443870 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:31.443858 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3317456a-b2d5-44e2-b346-8c2e42857626" containerName="revoke-keys" Jul 17 18:53:31.443870 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:31.443866 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3317456a-b2d5-44e2-b346-8c2e42857626" containerName="revoke-keys" Jul 17 18:53:31.447879 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:31.447862 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz" Jul 17 18:53:31.453369 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:31.453342 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz"] Jul 17 18:53:31.534154 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:31.534125 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0665c934-65bd-48cf-a4f8-2560a4788bd1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz\" (UID: \"0665c934-65bd-48cf-a4f8-2560a4788bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz" Jul 17 18:53:31.634828 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:31.634798 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0665c934-65bd-48cf-a4f8-2560a4788bd1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz\" (UID: \"0665c934-65bd-48cf-a4f8-2560a4788bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz" Jul 17 18:53:31.635417 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:31.635396 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0665c934-65bd-48cf-a4f8-2560a4788bd1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz\" (UID: \"0665c934-65bd-48cf-a4f8-2560a4788bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz" Jul 17 18:53:31.759714 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:31.759685 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz" Jul 17 18:53:31.881870 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:31.881781 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz"] Jul 17 18:53:31.884533 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:53:31.884510 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0665c934_65bd_48cf_a4f8_2560a4788bd1.slice/crio-b77d0530659698fae45f430a51a38598a7f2e457339dc2fd18bb24796a3fc861 WatchSource:0}: Error finding container b77d0530659698fae45f430a51a38598a7f2e457339dc2fd18bb24796a3fc861: Status 404 returned error can't find the container with id b77d0530659698fae45f430a51a38598a7f2e457339dc2fd18bb24796a3fc861 Jul 17 18:53:32.219382 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:32.219358 2572 scope.go:117] "RemoveContainer" containerID="68a7da39fbe4c7e3ebe578873e1c9c2e0623f00e2c7a0a58a5d49bfd437702da" Jul 17 18:53:32.305983 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:32.305956 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz" event={"ID":"0665c934-65bd-48cf-a4f8-2560a4788bd1","Type":"ContainerStarted","Data":"9550caf33a4804b9d1aabeeab2c18a14ca851aaa8c1d5759e5c29d81dff3aad5"} Jul 17 18:53:32.306111 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:32.306003 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz" event={"ID":"0665c934-65bd-48cf-a4f8-2560a4788bd1","Type":"ContainerStarted","Data":"b77d0530659698fae45f430a51a38598a7f2e457339dc2fd18bb24796a3fc861"} Jul 17 18:53:32.321017 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:32.320969 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz" podStartSLOduration=1.3209511649999999 podStartE2EDuration="1.320951165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:53:32.320811583 +0000 UTC m=+3407.730239210" watchObservedRunningTime="2026-07-17 18:53:32.320951165 +0000 UTC m=+3407.730378770" Jul 17 18:53:33.312295 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:33.312261 2572 generic.go:356] "Generic (PLEG): container finished" podID="0665c934-65bd-48cf-a4f8-2560a4788bd1" containerID="9550caf33a4804b9d1aabeeab2c18a14ca851aaa8c1d5759e5c29d81dff3aad5" exitCode=6 Jul 17 18:53:33.312734 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:33.312340 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz" event={"ID":"0665c934-65bd-48cf-a4f8-2560a4788bd1","Type":"ContainerDied","Data":"9550caf33a4804b9d1aabeeab2c18a14ca851aaa8c1d5759e5c29d81dff3aad5"} Jul 17 18:53:33.312734 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:33.312468 2572 scope.go:117] "RemoveContainer" containerID="9550caf33a4804b9d1aabeeab2c18a14ca851aaa8c1d5759e5c29d81dff3aad5" Jul 17 18:53:33.314228 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:33.314208 2572 generic.go:356] "Generic (PLEG): container finished" podID="78523e48-7ddf-40ec-ad38-e7afbeb77b55" containerID="9e5d43bfd224b38e38ca8a23ba984d4aef1ca883c1919351ba53f47f9d3ac397" exitCode=6 Jul 17 18:53:33.314323 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:33.314259 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28" event={"ID":"78523e48-7ddf-40ec-ad38-e7afbeb77b55","Type":"ContainerDied","Data":"9e5d43bfd224b38e38ca8a23ba984d4aef1ca883c1919351ba53f47f9d3ac397"} Jul 17 18:53:33.314323 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:33.314300 2572 scope.go:117] "RemoveContainer" containerID="68a7da39fbe4c7e3ebe578873e1c9c2e0623f00e2c7a0a58a5d49bfd437702da" Jul 17 18:53:33.314639 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:33.314607 2572 scope.go:117] "RemoveContainer" containerID="9e5d43bfd224b38e38ca8a23ba984d4aef1ca883c1919351ba53f47f9d3ac397" Jul 17 18:53:33.314845 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:53:33.314791 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28_odh-ai-gateway-infra(78523e48-7ddf-40ec-ad38-e7afbeb77b55)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28" podUID="78523e48-7ddf-40ec-ad38-e7afbeb77b55" Jul 17 18:53:34.005399 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:34.005371 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj"] Jul 17 18:53:34.005757 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:34.005744 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3317456a-b2d5-44e2-b346-8c2e42857626" containerName="revoke-keys" Jul 17 18:53:34.008625 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:34.008611 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj" Jul 17 18:53:34.015631 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:34.015437 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj"] Jul 17 18:53:34.157025 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:34.156970 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf722b6-c9d9-4935-8fd5-e9ece2762438-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj\" (UID: \"8cf722b6-c9d9-4935-8fd5-e9ece2762438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj" Jul 17 18:53:34.257944 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:34.257866 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf722b6-c9d9-4935-8fd5-e9ece2762438-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj\" (UID: \"8cf722b6-c9d9-4935-8fd5-e9ece2762438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj" Jul 17 18:53:34.258470 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:34.258452 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf722b6-c9d9-4935-8fd5-e9ece2762438-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj\" (UID: \"8cf722b6-c9d9-4935-8fd5-e9ece2762438\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj" Jul 17 18:53:34.319752 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:34.319725 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj" Jul 17 18:53:34.321989 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:34.321966 2572 generic.go:356] "Generic (PLEG): container finished" podID="0665c934-65bd-48cf-a4f8-2560a4788bd1" containerID="ae20e9f29145cc94f11aac43fd0c3e9813715bcb23502c56d2244e36d465a3db" exitCode=6 Jul 17 18:53:34.322107 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:34.322041 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz" event={"ID":"0665c934-65bd-48cf-a4f8-2560a4788bd1","Type":"ContainerDied","Data":"ae20e9f29145cc94f11aac43fd0c3e9813715bcb23502c56d2244e36d465a3db"} Jul 17 18:53:34.322107 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:34.322100 2572 scope.go:117] "RemoveContainer" containerID="9550caf33a4804b9d1aabeeab2c18a14ca851aaa8c1d5759e5c29d81dff3aad5" Jul 17 18:53:34.322313 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:34.322299 2572 scope.go:117] "RemoveContainer" containerID="ae20e9f29145cc94f11aac43fd0c3e9813715bcb23502c56d2244e36d465a3db" Jul 17 18:53:34.322499 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:53:34.322479 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz_odh-ai-gateway-infra(0665c934-65bd-48cf-a4f8-2560a4788bd1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz" podUID="0665c934-65bd-48cf-a4f8-2560a4788bd1" Jul 17 18:53:34.350310 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:34.350282 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28"] Jul 17 18:53:34.512268 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:34.512214 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28" Jul 17 18:53:34.658768 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:34.658694 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj"] Jul 17 18:53:34.661372 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:34.661351 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78523e48-7ddf-40ec-ad38-e7afbeb77b55-maas-api-service-ca\") pod \"78523e48-7ddf-40ec-ad38-e7afbeb77b55\" (UID: \"78523e48-7ddf-40ec-ad38-e7afbeb77b55\") " Jul 17 18:53:34.661597 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:53:34.661419 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf722b6_c9d9_4935_8fd5_e9ece2762438.slice/crio-0295653c3c6f35c2625eb2de3fcd92d5177905a077783ab2bb6217b9a05f9112 WatchSource:0}: Error finding container 0295653c3c6f35c2625eb2de3fcd92d5177905a077783ab2bb6217b9a05f9112: Status 404 returned error can't find the container with id 0295653c3c6f35c2625eb2de3fcd92d5177905a077783ab2bb6217b9a05f9112 Jul 17 18:53:34.661686 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:34.661666 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78523e48-7ddf-40ec-ad38-e7afbeb77b55-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78523e48-7ddf-40ec-ad38-e7afbeb77b55" (UID: "78523e48-7ddf-40ec-ad38-e7afbeb77b55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:53:34.762067 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:34.762031 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78523e48-7ddf-40ec-ad38-e7afbeb77b55-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:53:35.327961 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:35.327929 2572 generic.go:356] "Generic (PLEG): container finished" podID="8cf722b6-c9d9-4935-8fd5-e9ece2762438" containerID="56d2ef4a1bf3ac9f34c2be56fdd71ae294bb2fa88cd52f80d07ec441f028af40" exitCode=6 Jul 17 18:53:35.328348 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:35.327979 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj" event={"ID":"8cf722b6-c9d9-4935-8fd5-e9ece2762438","Type":"ContainerDied","Data":"56d2ef4a1bf3ac9f34c2be56fdd71ae294bb2fa88cd52f80d07ec441f028af40"} Jul 17 18:53:35.328348 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:35.328042 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj" event={"ID":"8cf722b6-c9d9-4935-8fd5-e9ece2762438","Type":"ContainerStarted","Data":"0295653c3c6f35c2625eb2de3fcd92d5177905a077783ab2bb6217b9a05f9112"} Jul 17 18:53:35.328348 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:35.328228 2572 scope.go:117] "RemoveContainer" containerID="56d2ef4a1bf3ac9f34c2be56fdd71ae294bb2fa88cd52f80d07ec441f028af40" Jul 17 18:53:35.329895 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:35.329880 2572 scope.go:117] "RemoveContainer" containerID="ae20e9f29145cc94f11aac43fd0c3e9813715bcb23502c56d2244e36d465a3db" Jul 17 18:53:35.330065 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:53:35.330048 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz_odh-ai-gateway-infra(0665c934-65bd-48cf-a4f8-2560a4788bd1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz" podUID="0665c934-65bd-48cf-a4f8-2560a4788bd1" Jul 17 18:53:35.330940 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:35.330919 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28" event={"ID":"78523e48-7ddf-40ec-ad38-e7afbeb77b55","Type":"ContainerDied","Data":"f686bfcb5e9ab087fde658e70a711e5bab9aabe23a2d9666998c2681afc83ce5"} Jul 17 18:53:35.331033 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:35.330953 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28" Jul 17 18:53:35.331033 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:35.330967 2572 scope.go:117] "RemoveContainer" containerID="9e5d43bfd224b38e38ca8a23ba984d4aef1ca883c1919351ba53f47f9d3ac397" Jul 17 18:53:35.356946 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:35.356921 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28"] Jul 17 18:53:35.362445 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:35.362424 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mfp28"] Jul 17 18:53:36.343151 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:36.343114 2572 generic.go:356] "Generic (PLEG): container finished" podID="8cf722b6-c9d9-4935-8fd5-e9ece2762438" containerID="d9db439b4e2df72690ec8379e68866884ea772b27160d9b48d5a4ff4061242ae" exitCode=6 Jul 17 18:53:36.343575 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:36.343160 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj" event={"ID":"8cf722b6-c9d9-4935-8fd5-e9ece2762438","Type":"ContainerDied","Data":"d9db439b4e2df72690ec8379e68866884ea772b27160d9b48d5a4ff4061242ae"} Jul 17 18:53:36.343575 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:36.343229 2572 scope.go:117] "RemoveContainer" containerID="56d2ef4a1bf3ac9f34c2be56fdd71ae294bb2fa88cd52f80d07ec441f028af40" Jul 17 18:53:36.343575 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:36.343426 2572 scope.go:117] "RemoveContainer" containerID="d9db439b4e2df72690ec8379e68866884ea772b27160d9b48d5a4ff4061242ae" Jul 17 18:53:36.343694 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:53:36.343636 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj_odh-ai-gateway-infra(8cf722b6-c9d9-4935-8fd5-e9ece2762438)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj" podUID="8cf722b6-c9d9-4935-8fd5-e9ece2762438" Jul 17 18:53:37.225115 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:37.225083 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78523e48-7ddf-40ec-ad38-e7afbeb77b55" path="/var/lib/kubelet/pods/78523e48-7ddf-40ec-ad38-e7afbeb77b55/volumes" Jul 17 18:53:37.348935 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:37.348911 2572 scope.go:117] "RemoveContainer" containerID="d9db439b4e2df72690ec8379e68866884ea772b27160d9b48d5a4ff4061242ae" Jul 17 18:53:37.349286 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:53:37.349131 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj_odh-ai-gateway-infra(8cf722b6-c9d9-4935-8fd5-e9ece2762438)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj" podUID="8cf722b6-c9d9-4935-8fd5-e9ece2762438" Jul 17 18:53:47.220387 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:47.220348 2572 scope.go:117] "RemoveContainer" containerID="ae20e9f29145cc94f11aac43fd0c3e9813715bcb23502c56d2244e36d465a3db" Jul 17 18:53:48.399727 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:48.399695 2572 generic.go:356] "Generic (PLEG): container finished" podID="0665c934-65bd-48cf-a4f8-2560a4788bd1" containerID="67dc4077831a9cff24cdf89c8d773a95949f8f6e57a2bc5ee5ac3415b8a29e97" exitCode=6 Jul 17 18:53:48.400129 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:48.399743 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz" event={"ID":"0665c934-65bd-48cf-a4f8-2560a4788bd1","Type":"ContainerDied","Data":"67dc4077831a9cff24cdf89c8d773a95949f8f6e57a2bc5ee5ac3415b8a29e97"} Jul 17 18:53:48.400129 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:48.399801 2572 scope.go:117] "RemoveContainer" containerID="ae20e9f29145cc94f11aac43fd0c3e9813715bcb23502c56d2244e36d465a3db" Jul 17 18:53:48.400129 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:48.400101 2572 scope.go:117] "RemoveContainer" containerID="67dc4077831a9cff24cdf89c8d773a95949f8f6e57a2bc5ee5ac3415b8a29e97" Jul 17 18:53:48.400322 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:53:48.400299 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz_odh-ai-gateway-infra(0665c934-65bd-48cf-a4f8-2560a4788bd1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz" podUID="0665c934-65bd-48cf-a4f8-2560a4788bd1" Jul 17 18:53:49.427693 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:49.427662 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz"] Jul 17 18:53:49.550536 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:49.550509 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz" Jul 17 18:53:49.589367 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:49.589341 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0665c934-65bd-48cf-a4f8-2560a4788bd1-maas-api-service-ca\") pod \"0665c934-65bd-48cf-a4f8-2560a4788bd1\" (UID: \"0665c934-65bd-48cf-a4f8-2560a4788bd1\") " Jul 17 18:53:49.589678 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:49.589656 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0665c934-65bd-48cf-a4f8-2560a4788bd1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0665c934-65bd-48cf-a4f8-2560a4788bd1" (UID: "0665c934-65bd-48cf-a4f8-2560a4788bd1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:53:49.690352 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:49.690279 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0665c934-65bd-48cf-a4f8-2560a4788bd1-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:53:50.411183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:50.411155 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz" Jul 17 18:53:50.411183 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:50.411175 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz" event={"ID":"0665c934-65bd-48cf-a4f8-2560a4788bd1","Type":"ContainerDied","Data":"b77d0530659698fae45f430a51a38598a7f2e457339dc2fd18bb24796a3fc861"} Jul 17 18:53:50.411434 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:50.411230 2572 scope.go:117] "RemoveContainer" containerID="67dc4077831a9cff24cdf89c8d773a95949f8f6e57a2bc5ee5ac3415b8a29e97" Jul 17 18:53:50.432063 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:50.432034 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz"] Jul 17 18:53:50.433537 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:50.433517 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2wpcz"] Jul 17 18:53:51.224248 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:51.224214 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0665c934-65bd-48cf-a4f8-2560a4788bd1" path="/var/lib/kubelet/pods/0665c934-65bd-48cf-a4f8-2560a4788bd1/volumes" Jul 17 18:53:51.224862 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:51.224847 2572 scope.go:117] "RemoveContainer" containerID="d9db439b4e2df72690ec8379e68866884ea772b27160d9b48d5a4ff4061242ae" Jul 17 18:53:52.422671 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:52.422641 2572 generic.go:356] "Generic (PLEG): container finished" podID="8cf722b6-c9d9-4935-8fd5-e9ece2762438" containerID="eb16fa1f9fb2d93353f2f6185f51594313d8d9e4542409b7653a60c9085842c0" exitCode=6 Jul 17 18:53:52.423129 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:52.422713 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj" event={"ID":"8cf722b6-c9d9-4935-8fd5-e9ece2762438","Type":"ContainerDied","Data":"eb16fa1f9fb2d93353f2f6185f51594313d8d9e4542409b7653a60c9085842c0"} Jul 17 18:53:52.423129 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:52.422776 2572 scope.go:117] "RemoveContainer" containerID="d9db439b4e2df72690ec8379e68866884ea772b27160d9b48d5a4ff4061242ae" Jul 17 18:53:52.423129 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:52.423081 2572 scope.go:117] "RemoveContainer" containerID="eb16fa1f9fb2d93353f2f6185f51594313d8d9e4542409b7653a60c9085842c0" Jul 17 18:53:52.423326 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:53:52.423308 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj_odh-ai-gateway-infra(8cf722b6-c9d9-4935-8fd5-e9ece2762438)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj" podUID="8cf722b6-c9d9-4935-8fd5-e9ece2762438" Jul 17 18:53:53.446533 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:53.446503 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj"] Jul 17 18:53:53.573600 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:53.573580 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj" Jul 17 18:53:53.620717 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:53.620683 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf722b6-c9d9-4935-8fd5-e9ece2762438-maas-api-service-ca\") pod \"8cf722b6-c9d9-4935-8fd5-e9ece2762438\" (UID: \"8cf722b6-c9d9-4935-8fd5-e9ece2762438\") " Jul 17 18:53:53.621003 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:53.620983 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf722b6-c9d9-4935-8fd5-e9ece2762438-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8cf722b6-c9d9-4935-8fd5-e9ece2762438" (UID: "8cf722b6-c9d9-4935-8fd5-e9ece2762438"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:53:53.722302 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:53.722218 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf722b6-c9d9-4935-8fd5-e9ece2762438-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:53:54.433858 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:54.433821 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj" event={"ID":"8cf722b6-c9d9-4935-8fd5-e9ece2762438","Type":"ContainerDied","Data":"0295653c3c6f35c2625eb2de3fcd92d5177905a077783ab2bb6217b9a05f9112"} Jul 17 18:53:54.434064 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:54.433882 2572 scope.go:117] "RemoveContainer" containerID="eb16fa1f9fb2d93353f2f6185f51594313d8d9e4542409b7653a60c9085842c0" Jul 17 18:53:54.434064 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:54.433840 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj" Jul 17 18:53:54.456577 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:54.456553 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj"] Jul 17 18:53:54.462626 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:54.462605 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-ndcxj"] Jul 17 18:53:55.224470 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:53:55.224443 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cf722b6-c9d9-4935-8fd5-e9ece2762438" path="/var/lib/kubelet/pods/8cf722b6-c9d9-4935-8fd5-e9ece2762438/volumes" Jul 17 18:54:05.873835 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.873801 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m"] Jul 17 18:54:05.874262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874123 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cf722b6-c9d9-4935-8fd5-e9ece2762438" containerName="revoke-keys" Jul 17 18:54:05.874262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874134 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cf722b6-c9d9-4935-8fd5-e9ece2762438" containerName="revoke-keys" Jul 17 18:54:05.874262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874145 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0665c934-65bd-48cf-a4f8-2560a4788bd1" containerName="revoke-keys" Jul 17 18:54:05.874262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874151 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0665c934-65bd-48cf-a4f8-2560a4788bd1" containerName="revoke-keys" Jul 17 18:54:05.874262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874159 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0665c934-65bd-48cf-a4f8-2560a4788bd1" containerName="revoke-keys" Jul 17 18:54:05.874262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874165 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0665c934-65bd-48cf-a4f8-2560a4788bd1" containerName="revoke-keys" Jul 17 18:54:05.874262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874173 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78523e48-7ddf-40ec-ad38-e7afbeb77b55" containerName="revoke-keys" Jul 17 18:54:05.874262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874179 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="78523e48-7ddf-40ec-ad38-e7afbeb77b55" containerName="revoke-keys" Jul 17 18:54:05.874262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874188 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78523e48-7ddf-40ec-ad38-e7afbeb77b55" containerName="revoke-keys" Jul 17 18:54:05.874262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874194 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="78523e48-7ddf-40ec-ad38-e7afbeb77b55" containerName="revoke-keys" Jul 17 18:54:05.874262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874200 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cf722b6-c9d9-4935-8fd5-e9ece2762438" containerName="revoke-keys" Jul 17 18:54:05.874262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874205 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cf722b6-c9d9-4935-8fd5-e9ece2762438" containerName="revoke-keys" Jul 17 18:54:05.874262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874252 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="78523e48-7ddf-40ec-ad38-e7afbeb77b55" containerName="revoke-keys" Jul 17 18:54:05.874262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874259 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cf722b6-c9d9-4935-8fd5-e9ece2762438" containerName="revoke-keys" Jul 17 18:54:05.874262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874264 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cf722b6-c9d9-4935-8fd5-e9ece2762438" containerName="revoke-keys" Jul 17 18:54:05.874262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874270 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0665c934-65bd-48cf-a4f8-2560a4788bd1" containerName="revoke-keys" Jul 17 18:54:05.874747 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874277 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="78523e48-7ddf-40ec-ad38-e7afbeb77b55" containerName="revoke-keys" Jul 17 18:54:05.874747 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874283 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="78523e48-7ddf-40ec-ad38-e7afbeb77b55" containerName="revoke-keys" Jul 17 18:54:05.874747 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874289 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0665c934-65bd-48cf-a4f8-2560a4788bd1" containerName="revoke-keys" Jul 17 18:54:05.874747 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.874293 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cf722b6-c9d9-4935-8fd5-e9ece2762438" containerName="revoke-keys" Jul 17 18:54:05.877271 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.877255 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m" Jul 17 18:54:05.879788 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.879766 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:54:05.879949 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.879787 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:54:05.885193 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.885164 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m"] Jul 17 18:54:05.929498 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:05.929473 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb5e747-4b2f-48c8-9d28-92a6280f12ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m\" (UID: \"ebb5e747-4b2f-48c8-9d28-92a6280f12ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m" Jul 17 18:54:06.030736 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:06.030694 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb5e747-4b2f-48c8-9d28-92a6280f12ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m\" (UID: \"ebb5e747-4b2f-48c8-9d28-92a6280f12ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m" Jul 17 18:54:06.031367 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:06.031342 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb5e747-4b2f-48c8-9d28-92a6280f12ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m\" (UID: \"ebb5e747-4b2f-48c8-9d28-92a6280f12ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m" Jul 17 18:54:06.188209 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:06.188121 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m" Jul 17 18:54:06.312445 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:06.312423 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m"] Jul 17 18:54:06.314525 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:54:06.314499 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb5e747_4b2f_48c8_9d28_92a6280f12ff.slice/crio-5a9dfdf1eeeafe448f1d0861fbc95af3065b7e337d35b43e3d2387f5d5bcabbc WatchSource:0}: Error finding container 5a9dfdf1eeeafe448f1d0861fbc95af3065b7e337d35b43e3d2387f5d5bcabbc: Status 404 returned error can't find the container with id 5a9dfdf1eeeafe448f1d0861fbc95af3065b7e337d35b43e3d2387f5d5bcabbc Jul 17 18:54:06.490297 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:06.490219 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m" event={"ID":"ebb5e747-4b2f-48c8-9d28-92a6280f12ff","Type":"ContainerStarted","Data":"5a9dfdf1eeeafe448f1d0861fbc95af3065b7e337d35b43e3d2387f5d5bcabbc"} Jul 17 18:54:07.495561 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:07.495529 2572 generic.go:356] "Generic (PLEG): container finished" podID="ebb5e747-4b2f-48c8-9d28-92a6280f12ff" containerID="697629972de4b08d85589ec0a5b4ad0c69ff549907ea868194bc0e0a6d877e9a" exitCode=6 Jul 17 18:54:07.495964 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:07.495576 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m" event={"ID":"ebb5e747-4b2f-48c8-9d28-92a6280f12ff","Type":"ContainerDied","Data":"697629972de4b08d85589ec0a5b4ad0c69ff549907ea868194bc0e0a6d877e9a"} Jul 17 18:54:07.495964 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:07.495782 2572 scope.go:117] "RemoveContainer" containerID="697629972de4b08d85589ec0a5b4ad0c69ff549907ea868194bc0e0a6d877e9a" Jul 17 18:54:08.002424 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.002349 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m"] Jul 17 18:54:08.002652 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.002641 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cf722b6-c9d9-4935-8fd5-e9ece2762438" containerName="revoke-keys" Jul 17 18:54:08.002699 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.002654 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cf722b6-c9d9-4935-8fd5-e9ece2762438" containerName="revoke-keys" Jul 17 18:54:08.002699 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.002676 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0665c934-65bd-48cf-a4f8-2560a4788bd1" containerName="revoke-keys" Jul 17 18:54:08.002699 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.002682 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0665c934-65bd-48cf-a4f8-2560a4788bd1" containerName="revoke-keys" Jul 17 18:54:08.002699 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.002690 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78523e48-7ddf-40ec-ad38-e7afbeb77b55" containerName="revoke-keys" Jul 17 18:54:08.002699 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.002695 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="78523e48-7ddf-40ec-ad38-e7afbeb77b55" containerName="revoke-keys" Jul 17 18:54:08.002851 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.002745 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0665c934-65bd-48cf-a4f8-2560a4788bd1" containerName="revoke-keys" Jul 17 18:54:08.005691 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.005674 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m" Jul 17 18:54:08.012501 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.012419 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m"] Jul 17 18:54:08.046272 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.046245 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd861277-9b86-4ed7-b054-49c9e85da935-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m\" (UID: \"dd861277-9b86-4ed7-b054-49c9e85da935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m" Jul 17 18:54:08.147492 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.147465 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd861277-9b86-4ed7-b054-49c9e85da935-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m\" (UID: \"dd861277-9b86-4ed7-b054-49c9e85da935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m" Jul 17 18:54:08.148097 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.148076 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd861277-9b86-4ed7-b054-49c9e85da935-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m\" (UID: \"dd861277-9b86-4ed7-b054-49c9e85da935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m" Jul 17 18:54:08.316687 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.316656 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m" Jul 17 18:54:08.442045 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.442020 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m"] Jul 17 18:54:08.444026 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:54:08.443989 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd861277_9b86_4ed7_b054_49c9e85da935.slice/crio-e4111fd63c075f4dbfde9d7ce55f1cbaf02ce1513b15610d0e97926a8116325c WatchSource:0}: Error finding container e4111fd63c075f4dbfde9d7ce55f1cbaf02ce1513b15610d0e97926a8116325c: Status 404 returned error can't find the container with id e4111fd63c075f4dbfde9d7ce55f1cbaf02ce1513b15610d0e97926a8116325c Jul 17 18:54:08.501215 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.501190 2572 generic.go:356] "Generic (PLEG): container finished" podID="ebb5e747-4b2f-48c8-9d28-92a6280f12ff" containerID="677386ac521ab37b5860e848c4ca56ce8704d60584a218ec9a00a6e6fc9f948a" exitCode=6 Jul 17 18:54:08.501573 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.501261 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m" event={"ID":"ebb5e747-4b2f-48c8-9d28-92a6280f12ff","Type":"ContainerDied","Data":"677386ac521ab37b5860e848c4ca56ce8704d60584a218ec9a00a6e6fc9f948a"} Jul 17 18:54:08.501573 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.501319 2572 scope.go:117] "RemoveContainer" containerID="697629972de4b08d85589ec0a5b4ad0c69ff549907ea868194bc0e0a6d877e9a" Jul 17 18:54:08.501573 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.501505 2572 scope.go:117] "RemoveContainer" containerID="677386ac521ab37b5860e848c4ca56ce8704d60584a218ec9a00a6e6fc9f948a" Jul 17 18:54:08.501725 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:54:08.501706 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m_odh-ai-gateway-infra(ebb5e747-4b2f-48c8-9d28-92a6280f12ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m" podUID="ebb5e747-4b2f-48c8-9d28-92a6280f12ff" Jul 17 18:54:08.502604 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:08.502394 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m" event={"ID":"dd861277-9b86-4ed7-b054-49c9e85da935","Type":"ContainerStarted","Data":"e4111fd63c075f4dbfde9d7ce55f1cbaf02ce1513b15610d0e97926a8116325c"} Jul 17 18:54:09.508743 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:09.508652 2572 generic.go:356] "Generic (PLEG): container finished" podID="dd861277-9b86-4ed7-b054-49c9e85da935" containerID="99a2d08c5d5cdddf569673c78fadb1804cc4fc9f6fed138a0db515fb7e390449" exitCode=6 Jul 17 18:54:09.508743 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:09.508712 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m" event={"ID":"dd861277-9b86-4ed7-b054-49c9e85da935","Type":"ContainerDied","Data":"99a2d08c5d5cdddf569673c78fadb1804cc4fc9f6fed138a0db515fb7e390449"} Jul 17 18:54:09.509261 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:09.508970 2572 scope.go:117] "RemoveContainer" containerID="99a2d08c5d5cdddf569673c78fadb1804cc4fc9f6fed138a0db515fb7e390449" Jul 17 18:54:09.510566 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:09.510551 2572 scope.go:117] "RemoveContainer" containerID="677386ac521ab37b5860e848c4ca56ce8704d60584a218ec9a00a6e6fc9f948a" Jul 17 18:54:09.510718 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:54:09.510702 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m_odh-ai-gateway-infra(ebb5e747-4b2f-48c8-9d28-92a6280f12ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m" podUID="ebb5e747-4b2f-48c8-9d28-92a6280f12ff" Jul 17 18:54:10.517141 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:10.517041 2572 generic.go:356] "Generic (PLEG): container finished" podID="dd861277-9b86-4ed7-b054-49c9e85da935" containerID="d5149167775a6b3bda64614aac88401cf469d6787f924635b9988b1801309159" exitCode=6 Jul 17 18:54:10.517141 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:10.517095 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m" event={"ID":"dd861277-9b86-4ed7-b054-49c9e85da935","Type":"ContainerDied","Data":"d5149167775a6b3bda64614aac88401cf469d6787f924635b9988b1801309159"} Jul 17 18:54:10.517625 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:10.517150 2572 scope.go:117] "RemoveContainer" containerID="99a2d08c5d5cdddf569673c78fadb1804cc4fc9f6fed138a0db515fb7e390449" Jul 17 18:54:10.517625 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:10.517298 2572 scope.go:117] "RemoveContainer" containerID="d5149167775a6b3bda64614aac88401cf469d6787f924635b9988b1801309159" Jul 17 18:54:10.517625 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:54:10.517513 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m_odh-ai-gateway-infra(dd861277-9b86-4ed7-b054-49c9e85da935)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m" podUID="dd861277-9b86-4ed7-b054-49c9e85da935" Jul 17 18:54:11.405062 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:11.405026 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs"] Jul 17 18:54:11.523393 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:11.523364 2572 scope.go:117] "RemoveContainer" containerID="d5149167775a6b3bda64614aac88401cf469d6787f924635b9988b1801309159" Jul 17 18:54:11.523786 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:54:11.523567 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m_odh-ai-gateway-infra(dd861277-9b86-4ed7-b054-49c9e85da935)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m" podUID="dd861277-9b86-4ed7-b054-49c9e85da935" Jul 17 18:54:21.220408 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:21.220375 2572 scope.go:117] "RemoveContainer" containerID="677386ac521ab37b5860e848c4ca56ce8704d60584a218ec9a00a6e6fc9f948a" Jul 17 18:54:21.487256 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:21.487188 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8"] Jul 17 18:54:21.490376 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:21.490360 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8" Jul 17 18:54:21.499368 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:21.499345 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8"] Jul 17 18:54:21.567640 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:21.567610 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m" event={"ID":"ebb5e747-4b2f-48c8-9d28-92a6280f12ff","Type":"ContainerStarted","Data":"02d5df36df8b811b4208252a2ab9f475e9ed7f2482bf9d2c33a8a2999d274624"} Jul 17 18:54:21.570902 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:21.570879 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbccdb2-974b-4870-8d62-b91e2d1d77f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8\" (UID: \"fbbccdb2-974b-4870-8d62-b91e2d1d77f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8" Jul 17 18:54:21.583298 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:21.583253 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m" podStartSLOduration=16.583240662 podStartE2EDuration="16.583240662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:54:21.582139003 +0000 UTC m=+3456.991566610" watchObservedRunningTime="2026-07-17 18:54:21.583240662 +0000 UTC m=+3456.992668318" Jul 17 18:54:21.671813 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:21.671781 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbccdb2-974b-4870-8d62-b91e2d1d77f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8\" (UID: \"fbbccdb2-974b-4870-8d62-b91e2d1d77f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8" Jul 17 18:54:21.672430 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:21.672411 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbccdb2-974b-4870-8d62-b91e2d1d77f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8\" (UID: \"fbbccdb2-974b-4870-8d62-b91e2d1d77f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8" Jul 17 18:54:21.801110 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:21.801079 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8" Jul 17 18:54:21.928973 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:21.928947 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8"] Jul 17 18:54:21.931174 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:54:21.931147 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbccdb2_974b_4870_8d62_b91e2d1d77f7.slice/crio-b854bbcfa3907a189b0383f5bd24b14cc549ce83d64123a938fc23f9e7b23108 WatchSource:0}: Error finding container b854bbcfa3907a189b0383f5bd24b14cc549ce83d64123a938fc23f9e7b23108: Status 404 returned error can't find the container with id b854bbcfa3907a189b0383f5bd24b14cc549ce83d64123a938fc23f9e7b23108 Jul 17 18:54:22.573352 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:22.573314 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8" event={"ID":"fbbccdb2-974b-4870-8d62-b91e2d1d77f7","Type":"ContainerStarted","Data":"e93660064384029c869b4aa70c4e0e2d1b6d05ef750d36cc962f014360575169"} Jul 17 18:54:22.573808 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:22.573398 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8" event={"ID":"fbbccdb2-974b-4870-8d62-b91e2d1d77f7","Type":"ContainerStarted","Data":"b854bbcfa3907a189b0383f5bd24b14cc549ce83d64123a938fc23f9e7b23108"} Jul 17 18:54:22.575096 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:22.575073 2572 generic.go:356] "Generic (PLEG): container finished" podID="ebb5e747-4b2f-48c8-9d28-92a6280f12ff" containerID="02d5df36df8b811b4208252a2ab9f475e9ed7f2482bf9d2c33a8a2999d274624" exitCode=6 Jul 17 18:54:22.575199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:22.575141 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m" event={"ID":"ebb5e747-4b2f-48c8-9d28-92a6280f12ff","Type":"ContainerDied","Data":"02d5df36df8b811b4208252a2ab9f475e9ed7f2482bf9d2c33a8a2999d274624"} Jul 17 18:54:22.575199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:22.575194 2572 scope.go:117] "RemoveContainer" containerID="677386ac521ab37b5860e848c4ca56ce8704d60584a218ec9a00a6e6fc9f948a" Jul 17 18:54:22.575481 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:22.575464 2572 scope.go:117] "RemoveContainer" containerID="02d5df36df8b811b4208252a2ab9f475e9ed7f2482bf9d2c33a8a2999d274624" Jul 17 18:54:22.575679 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:54:22.575660 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m_odh-ai-gateway-infra(ebb5e747-4b2f-48c8-9d28-92a6280f12ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m" podUID="ebb5e747-4b2f-48c8-9d28-92a6280f12ff" Jul 17 18:54:22.588897 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:22.588857 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8" podStartSLOduration=1.588844344 podStartE2EDuration="1.588844344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:54:22.587044649 +0000 UTC m=+3457.996472280" watchObservedRunningTime="2026-07-17 18:54:22.588844344 +0000 UTC m=+3457.998271959" Jul 17 18:54:22.593637 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:22.593616 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m"] Jul 17 18:54:23.582120 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:23.582094 2572 generic.go:356] "Generic (PLEG): container finished" podID="fbbccdb2-974b-4870-8d62-b91e2d1d77f7" containerID="e93660064384029c869b4aa70c4e0e2d1b6d05ef750d36cc962f014360575169" exitCode=6 Jul 17 18:54:23.582461 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:23.582174 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8" event={"ID":"fbbccdb2-974b-4870-8d62-b91e2d1d77f7","Type":"ContainerDied","Data":"e93660064384029c869b4aa70c4e0e2d1b6d05ef750d36cc962f014360575169"} Jul 17 18:54:23.582461 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:23.582424 2572 scope.go:117] "RemoveContainer" containerID="e93660064384029c869b4aa70c4e0e2d1b6d05ef750d36cc962f014360575169" Jul 17 18:54:23.704315 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:23.704288 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m" Jul 17 18:54:23.788460 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:23.788427 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb5e747-4b2f-48c8-9d28-92a6280f12ff-maas-api-service-ca\") pod \"ebb5e747-4b2f-48c8-9d28-92a6280f12ff\" (UID: \"ebb5e747-4b2f-48c8-9d28-92a6280f12ff\") " Jul 17 18:54:23.788805 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:23.788785 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb5e747-4b2f-48c8-9d28-92a6280f12ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ebb5e747-4b2f-48c8-9d28-92a6280f12ff" (UID: "ebb5e747-4b2f-48c8-9d28-92a6280f12ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:54:23.889965 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:23.889873 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebb5e747-4b2f-48c8-9d28-92a6280f12ff-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:54:24.219871 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:24.219849 2572 scope.go:117] "RemoveContainer" containerID="d5149167775a6b3bda64614aac88401cf469d6787f924635b9988b1801309159" Jul 17 18:54:24.588345 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:24.588309 2572 generic.go:356] "Generic (PLEG): container finished" podID="fbbccdb2-974b-4870-8d62-b91e2d1d77f7" containerID="b2c5fcfceeb48fda29b628f041a3ce399ec3bd519b98b7fe5d5ecb47914d7ae3" exitCode=6 Jul 17 18:54:24.588790 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:24.588395 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8" event={"ID":"fbbccdb2-974b-4870-8d62-b91e2d1d77f7","Type":"ContainerDied","Data":"b2c5fcfceeb48fda29b628f041a3ce399ec3bd519b98b7fe5d5ecb47914d7ae3"} Jul 17 18:54:24.588790 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:24.588456 2572 scope.go:117] "RemoveContainer" containerID="e93660064384029c869b4aa70c4e0e2d1b6d05ef750d36cc962f014360575169" Jul 17 18:54:24.588790 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:24.588618 2572 scope.go:117] "RemoveContainer" containerID="b2c5fcfceeb48fda29b628f041a3ce399ec3bd519b98b7fe5d5ecb47914d7ae3" Jul 17 18:54:24.588936 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:54:24.588854 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8_odh-ai-gateway-infra(fbbccdb2-974b-4870-8d62-b91e2d1d77f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8" podUID="fbbccdb2-974b-4870-8d62-b91e2d1d77f7" Jul 17 18:54:24.590284 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:24.590263 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m" event={"ID":"dd861277-9b86-4ed7-b054-49c9e85da935","Type":"ContainerStarted","Data":"3aea3c00163e7f9a25fdba86bf6eaf6cb9f185e124ce9248e11c6df70522263e"} Jul 17 18:54:24.591374 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:24.591353 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m" event={"ID":"ebb5e747-4b2f-48c8-9d28-92a6280f12ff","Type":"ContainerDied","Data":"5a9dfdf1eeeafe448f1d0861fbc95af3065b7e337d35b43e3d2387f5d5bcabbc"} Jul 17 18:54:24.591374 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:24.591370 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m" Jul 17 18:54:24.599485 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:24.599470 2572 scope.go:117] "RemoveContainer" containerID="02d5df36df8b811b4208252a2ab9f475e9ed7f2482bf9d2c33a8a2999d274624" Jul 17 18:54:24.617441 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:24.617420 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m"] Jul 17 18:54:24.621269 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:24.621250 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-qgh9m"] Jul 17 18:54:24.634461 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:24.634419 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m" podStartSLOduration=17.634404899 podStartE2EDuration="17.634404899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:54:24.633349341 +0000 UTC m=+3460.042776971" watchObservedRunningTime="2026-07-17 18:54:24.634404899 +0000 UTC m=+3460.043832551" Jul 17 18:54:25.224285 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:25.224202 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebb5e747-4b2f-48c8-9d28-92a6280f12ff" path="/var/lib/kubelet/pods/ebb5e747-4b2f-48c8-9d28-92a6280f12ff/volumes" Jul 17 18:54:25.597588 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:25.597558 2572 scope.go:117] "RemoveContainer" containerID="b2c5fcfceeb48fda29b628f041a3ce399ec3bd519b98b7fe5d5ecb47914d7ae3" Jul 17 18:54:25.598064 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:54:25.597806 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8_odh-ai-gateway-infra(fbbccdb2-974b-4870-8d62-b91e2d1d77f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8" podUID="fbbccdb2-974b-4870-8d62-b91e2d1d77f7" Jul 17 18:54:25.598970 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:25.598948 2572 generic.go:356] "Generic (PLEG): container finished" podID="dd861277-9b86-4ed7-b054-49c9e85da935" containerID="3aea3c00163e7f9a25fdba86bf6eaf6cb9f185e124ce9248e11c6df70522263e" exitCode=6 Jul 17 18:54:25.599061 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:25.599032 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m" event={"ID":"dd861277-9b86-4ed7-b054-49c9e85da935","Type":"ContainerDied","Data":"3aea3c00163e7f9a25fdba86bf6eaf6cb9f185e124ce9248e11c6df70522263e"} Jul 17 18:54:25.599119 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:25.599082 2572 scope.go:117] "RemoveContainer" containerID="d5149167775a6b3bda64614aac88401cf469d6787f924635b9988b1801309159" Jul 17 18:54:25.599383 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:25.599366 2572 scope.go:117] "RemoveContainer" containerID="3aea3c00163e7f9a25fdba86bf6eaf6cb9f185e124ce9248e11c6df70522263e" Jul 17 18:54:25.599579 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:54:25.599558 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m_odh-ai-gateway-infra(dd861277-9b86-4ed7-b054-49c9e85da935)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m" podUID="dd861277-9b86-4ed7-b054-49c9e85da935" Jul 17 18:54:25.642568 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:25.642531 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m"] Jul 17 18:54:26.425475 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.425425 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" podUID="851e200f-6bbb-48ae-8f88-dd942f80f5c9" containerName="main" containerID="cri-o://735b66fcb6cef492e8fc75877a38b2acfff321491e7120349588775d504823d8" gracePeriod=45 Jul 17 18:54:26.607189 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.607155 2572 generic.go:356] "Generic (PLEG): container finished" podID="851e200f-6bbb-48ae-8f88-dd942f80f5c9" containerID="735b66fcb6cef492e8fc75877a38b2acfff321491e7120349588775d504823d8" exitCode=0 Jul 17 18:54:26.607646 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.607233 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" event={"ID":"851e200f-6bbb-48ae-8f88-dd942f80f5c9","Type":"ContainerDied","Data":"735b66fcb6cef492e8fc75877a38b2acfff321491e7120349588775d504823d8"} Jul 17 18:54:26.678440 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.678383 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:54:26.751053 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.751028 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m" Jul 17 18:54:26.813735 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.813700 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-home\") pod \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " Jul 17 18:54:26.813735 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.813737 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-tmp-dir\") pod \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " Jul 17 18:54:26.813960 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.813784 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851e200f-6bbb-48ae-8f88-dd942f80f5c9-tls-certs\") pod \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " Jul 17 18:54:26.813960 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.813820 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-dshm\") pod \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " Jul 17 18:54:26.813960 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.813843 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsz25\" (UniqueName: \"kubernetes.io/projected/851e200f-6bbb-48ae-8f88-dd942f80f5c9-kube-api-access-gsz25\") pod \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " Jul 17 18:54:26.813960 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.813869 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-model-cache\") pod \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\" (UID: \"851e200f-6bbb-48ae-8f88-dd942f80f5c9\") " Jul 17 18:54:26.814184 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.814000 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-home" (OuterVolumeSpecName: "home") pod "851e200f-6bbb-48ae-8f88-dd942f80f5c9" (UID: "851e200f-6bbb-48ae-8f88-dd942f80f5c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:54:26.814184 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.814071 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "851e200f-6bbb-48ae-8f88-dd942f80f5c9" (UID: "851e200f-6bbb-48ae-8f88-dd942f80f5c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:54:26.814184 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.814154 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-home\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:54:26.814293 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.814198 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-tmp-dir\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:54:26.814293 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.814210 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "851e200f-6bbb-48ae-8f88-dd942f80f5c9" (UID: "851e200f-6bbb-48ae-8f88-dd942f80f5c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:54:26.816123 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.816090 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-dshm" (OuterVolumeSpecName: "dshm") pod "851e200f-6bbb-48ae-8f88-dd942f80f5c9" (UID: "851e200f-6bbb-48ae-8f88-dd942f80f5c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:54:26.816123 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.816110 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851e200f-6bbb-48ae-8f88-dd942f80f5c9-kube-api-access-gsz25" (OuterVolumeSpecName: "kube-api-access-gsz25") pod "851e200f-6bbb-48ae-8f88-dd942f80f5c9" (UID: "851e200f-6bbb-48ae-8f88-dd942f80f5c9"). InnerVolumeSpecName "kube-api-access-gsz25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:54:26.816247 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.816138 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/851e200f-6bbb-48ae-8f88-dd942f80f5c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "851e200f-6bbb-48ae-8f88-dd942f80f5c9" (UID: "851e200f-6bbb-48ae-8f88-dd942f80f5c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:54:26.914732 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.914682 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd861277-9b86-4ed7-b054-49c9e85da935-maas-api-service-ca\") pod \"dd861277-9b86-4ed7-b054-49c9e85da935\" (UID: \"dd861277-9b86-4ed7-b054-49c9e85da935\") " Jul 17 18:54:26.915040 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.914971 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/851e200f-6bbb-48ae-8f88-dd942f80f5c9-tls-certs\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:54:26.915100 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.915048 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-dshm\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:54:26.915100 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.915063 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsz25\" (UniqueName: \"kubernetes.io/projected/851e200f-6bbb-48ae-8f88-dd942f80f5c9-kube-api-access-gsz25\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:54:26.915100 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.915076 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/851e200f-6bbb-48ae-8f88-dd942f80f5c9-model-cache\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:54:26.915204 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:26.915121 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd861277-9b86-4ed7-b054-49c9e85da935-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd861277-9b86-4ed7-b054-49c9e85da935" (UID: "dd861277-9b86-4ed7-b054-49c9e85da935"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:54:27.016134 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:27.016046 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd861277-9b86-4ed7-b054-49c9e85da935-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:54:27.614488 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:27.614456 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" Jul 17 18:54:27.614970 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:27.614488 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs" event={"ID":"851e200f-6bbb-48ae-8f88-dd942f80f5c9","Type":"ContainerDied","Data":"43cccd11c4aab78d31e3dbdb0834f85754da62afcb190142ac7d88e0ed2b47b5"} Jul 17 18:54:27.614970 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:27.614540 2572 scope.go:117] "RemoveContainer" containerID="735b66fcb6cef492e8fc75877a38b2acfff321491e7120349588775d504823d8" Jul 17 18:54:27.615881 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:27.615844 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m" event={"ID":"dd861277-9b86-4ed7-b054-49c9e85da935","Type":"ContainerDied","Data":"e4111fd63c075f4dbfde9d7ce55f1cbaf02ce1513b15610d0e97926a8116325c"} Jul 17 18:54:27.615958 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:27.615940 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m" Jul 17 18:54:27.623261 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:27.623242 2572 scope.go:117] "RemoveContainer" containerID="3aea3c00163e7f9a25fdba86bf6eaf6cb9f185e124ce9248e11c6df70522263e" Jul 17 18:54:27.632930 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:27.632909 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs"] Jul 17 18:54:27.636079 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:27.636056 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-52fc47/auth-test-model-52fc47-kserve-d589fff6c-6sbhs"] Jul 17 18:54:27.645952 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:27.645910 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m"] Jul 17 18:54:27.647554 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:27.647529 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-6d74m"] Jul 17 18:54:29.224806 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:29.224777 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="851e200f-6bbb-48ae-8f88-dd942f80f5c9" path="/var/lib/kubelet/pods/851e200f-6bbb-48ae-8f88-dd942f80f5c9/volumes" Jul 17 18:54:29.225190 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:29.225102 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd861277-9b86-4ed7-b054-49c9e85da935" path="/var/lib/kubelet/pods/dd861277-9b86-4ed7-b054-49c9e85da935/volumes" Jul 17 18:54:34.050392 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050356 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8"] Jul 17 18:54:34.050777 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050665 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851e200f-6bbb-48ae-8f88-dd942f80f5c9" containerName="main" Jul 17 18:54:34.050777 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050675 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="851e200f-6bbb-48ae-8f88-dd942f80f5c9" containerName="main" Jul 17 18:54:34.050777 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050684 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd861277-9b86-4ed7-b054-49c9e85da935" containerName="revoke-keys" Jul 17 18:54:34.050777 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050689 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd861277-9b86-4ed7-b054-49c9e85da935" containerName="revoke-keys" Jul 17 18:54:34.050777 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050696 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd861277-9b86-4ed7-b054-49c9e85da935" containerName="revoke-keys" Jul 17 18:54:34.050777 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050702 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd861277-9b86-4ed7-b054-49c9e85da935" containerName="revoke-keys" Jul 17 18:54:34.050777 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050708 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb5e747-4b2f-48c8-9d28-92a6280f12ff" containerName="revoke-keys" Jul 17 18:54:34.050777 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050712 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb5e747-4b2f-48c8-9d28-92a6280f12ff" containerName="revoke-keys" Jul 17 18:54:34.050777 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050726 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb5e747-4b2f-48c8-9d28-92a6280f12ff" containerName="revoke-keys" Jul 17 18:54:34.050777 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050731 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb5e747-4b2f-48c8-9d28-92a6280f12ff" containerName="revoke-keys" Jul 17 18:54:34.050777 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050739 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb5e747-4b2f-48c8-9d28-92a6280f12ff" containerName="revoke-keys" Jul 17 18:54:34.050777 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050744 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb5e747-4b2f-48c8-9d28-92a6280f12ff" containerName="revoke-keys" Jul 17 18:54:34.051168 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050803 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd861277-9b86-4ed7-b054-49c9e85da935" containerName="revoke-keys" Jul 17 18:54:34.051168 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050811 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="851e200f-6bbb-48ae-8f88-dd942f80f5c9" containerName="main" Jul 17 18:54:34.051168 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050817 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb5e747-4b2f-48c8-9d28-92a6280f12ff" containerName="revoke-keys" Jul 17 18:54:34.051168 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050823 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb5e747-4b2f-48c8-9d28-92a6280f12ff" containerName="revoke-keys" Jul 17 18:54:34.051168 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050828 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb5e747-4b2f-48c8-9d28-92a6280f12ff" containerName="revoke-keys" Jul 17 18:54:34.051168 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.050834 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd861277-9b86-4ed7-b054-49c9e85da935" containerName="revoke-keys" Jul 17 18:54:34.055037 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.054999 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8" Jul 17 18:54:34.061098 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.061073 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8"] Jul 17 18:54:34.075731 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.075702 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b652d0ce-f8b5-4239-84f9-6d5227fb2b7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-vttb8\" (UID: \"b652d0ce-f8b5-4239-84f9-6d5227fb2b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8" Jul 17 18:54:34.176620 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.176590 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b652d0ce-f8b5-4239-84f9-6d5227fb2b7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-vttb8\" (UID: \"b652d0ce-f8b5-4239-84f9-6d5227fb2b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8" Jul 17 18:54:34.177255 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.177236 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b652d0ce-f8b5-4239-84f9-6d5227fb2b7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-vttb8\" (UID: \"b652d0ce-f8b5-4239-84f9-6d5227fb2b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8" Jul 17 18:54:34.366920 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.366841 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8" Jul 17 18:54:34.492976 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.492945 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8"] Jul 17 18:54:34.494984 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:54:34.494960 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb652d0ce_f8b5_4239_84f9_6d5227fb2b7b.slice/crio-017e6a8ce19ef9b3684f5ed24e057b3c772c70bc968761044ba969fc437bc612 WatchSource:0}: Error finding container 017e6a8ce19ef9b3684f5ed24e057b3c772c70bc968761044ba969fc437bc612: Status 404 returned error can't find the container with id 017e6a8ce19ef9b3684f5ed24e057b3c772c70bc968761044ba969fc437bc612 Jul 17 18:54:34.648925 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:34.648897 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8" event={"ID":"b652d0ce-f8b5-4239-84f9-6d5227fb2b7b","Type":"ContainerStarted","Data":"017e6a8ce19ef9b3684f5ed24e057b3c772c70bc968761044ba969fc437bc612"} Jul 17 18:54:35.654598 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:35.654560 2572 generic.go:356] "Generic (PLEG): container finished" podID="b652d0ce-f8b5-4239-84f9-6d5227fb2b7b" containerID="c386a8a1ec930aaf4727c7926ff96b45dec04a10d3662e958b590b75e6ce37fb" exitCode=6 Jul 17 18:54:35.655042 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:35.654642 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8" event={"ID":"b652d0ce-f8b5-4239-84f9-6d5227fb2b7b","Type":"ContainerDied","Data":"c386a8a1ec930aaf4727c7926ff96b45dec04a10d3662e958b590b75e6ce37fb"} Jul 17 18:54:35.655042 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:35.654833 2572 scope.go:117] "RemoveContainer" containerID="c386a8a1ec930aaf4727c7926ff96b45dec04a10d3662e958b590b75e6ce37fb" Jul 17 18:54:36.660445 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:36.660412 2572 generic.go:356] "Generic (PLEG): container finished" podID="b652d0ce-f8b5-4239-84f9-6d5227fb2b7b" containerID="0c868b6378f2c256127b949e6743e6ebc20203822660aca2678d7fb892dde572" exitCode=6 Jul 17 18:54:36.660945 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:36.660451 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8" event={"ID":"b652d0ce-f8b5-4239-84f9-6d5227fb2b7b","Type":"ContainerDied","Data":"0c868b6378f2c256127b949e6743e6ebc20203822660aca2678d7fb892dde572"} Jul 17 18:54:36.660945 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:36.660514 2572 scope.go:117] "RemoveContainer" containerID="c386a8a1ec930aaf4727c7926ff96b45dec04a10d3662e958b590b75e6ce37fb" Jul 17 18:54:36.660945 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:36.660697 2572 scope.go:117] "RemoveContainer" containerID="0c868b6378f2c256127b949e6743e6ebc20203822660aca2678d7fb892dde572" Jul 17 18:54:36.661157 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:54:36.660953 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-vttb8_odh-ai-gateway-infra(b652d0ce-f8b5-4239-84f9-6d5227fb2b7b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8" podUID="b652d0ce-f8b5-4239-84f9-6d5227fb2b7b" Jul 17 18:54:37.666528 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:37.666492 2572 scope.go:117] "RemoveContainer" containerID="0c868b6378f2c256127b949e6743e6ebc20203822660aca2678d7fb892dde572" Jul 17 18:54:37.666924 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:54:37.666682 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-vttb8_odh-ai-gateway-infra(b652d0ce-f8b5-4239-84f9-6d5227fb2b7b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8" podUID="b652d0ce-f8b5-4239-84f9-6d5227fb2b7b" Jul 17 18:54:41.220187 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:41.220157 2572 scope.go:117] "RemoveContainer" containerID="b2c5fcfceeb48fda29b628f041a3ce399ec3bd519b98b7fe5d5ecb47914d7ae3" Jul 17 18:54:41.687161 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:41.687124 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8" event={"ID":"fbbccdb2-974b-4870-8d62-b91e2d1d77f7","Type":"ContainerStarted","Data":"997a98afb15c05dbab837345ef0de77786a704419c681e9cd7a5ceef80deb902"} Jul 17 18:54:42.693366 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:42.693332 2572 generic.go:356] "Generic (PLEG): container finished" podID="fbbccdb2-974b-4870-8d62-b91e2d1d77f7" containerID="997a98afb15c05dbab837345ef0de77786a704419c681e9cd7a5ceef80deb902" exitCode=6 Jul 17 18:54:42.693802 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:42.693407 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8" event={"ID":"fbbccdb2-974b-4870-8d62-b91e2d1d77f7","Type":"ContainerDied","Data":"997a98afb15c05dbab837345ef0de77786a704419c681e9cd7a5ceef80deb902"} Jul 17 18:54:42.693802 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:42.693478 2572 scope.go:117] "RemoveContainer" containerID="b2c5fcfceeb48fda29b628f041a3ce399ec3bd519b98b7fe5d5ecb47914d7ae3" Jul 17 18:54:42.693802 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:42.693798 2572 scope.go:117] "RemoveContainer" containerID="997a98afb15c05dbab837345ef0de77786a704419c681e9cd7a5ceef80deb902" Jul 17 18:54:42.694082 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:54:42.694059 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8_odh-ai-gateway-infra(fbbccdb2-974b-4870-8d62-b91e2d1d77f7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8" podUID="fbbccdb2-974b-4870-8d62-b91e2d1d77f7" Jul 17 18:54:42.712050 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:42.712026 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8"] Jul 17 18:54:43.830400 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:43.830379 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8" Jul 17 18:54:43.849239 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:43.849214 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbccdb2-974b-4870-8d62-b91e2d1d77f7-maas-api-service-ca\") pod \"fbbccdb2-974b-4870-8d62-b91e2d1d77f7\" (UID: \"fbbccdb2-974b-4870-8d62-b91e2d1d77f7\") " Jul 17 18:54:43.849659 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:43.849620 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbbccdb2-974b-4870-8d62-b91e2d1d77f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fbbccdb2-974b-4870-8d62-b91e2d1d77f7" (UID: "fbbccdb2-974b-4870-8d62-b91e2d1d77f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:54:43.950292 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:43.950208 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbbccdb2-974b-4870-8d62-b91e2d1d77f7-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:54:44.705334 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:44.705300 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8" Jul 17 18:54:44.705547 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:44.705339 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8" event={"ID":"fbbccdb2-974b-4870-8d62-b91e2d1d77f7","Type":"ContainerDied","Data":"b854bbcfa3907a189b0383f5bd24b14cc549ce83d64123a938fc23f9e7b23108"} Jul 17 18:54:44.705547 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:44.705415 2572 scope.go:117] "RemoveContainer" containerID="997a98afb15c05dbab837345ef0de77786a704419c681e9cd7a5ceef80deb902" Jul 17 18:54:44.729209 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:44.729181 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8"] Jul 17 18:54:44.733102 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:44.733078 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-h9fj8"] Jul 17 18:54:45.225831 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:45.225797 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbbccdb2-974b-4870-8d62-b91e2d1d77f7" path="/var/lib/kubelet/pods/fbbccdb2-974b-4870-8d62-b91e2d1d77f7/volumes" Jul 17 18:54:48.220129 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:48.220099 2572 scope.go:117] "RemoveContainer" containerID="0c868b6378f2c256127b949e6743e6ebc20203822660aca2678d7fb892dde572" Jul 17 18:54:48.727424 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:48.727389 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8" event={"ID":"b652d0ce-f8b5-4239-84f9-6d5227fb2b7b","Type":"ContainerStarted","Data":"bac4a7311cab335a478f7df8984cd139b6d4019eca908c01d258c8462f0099fa"} Jul 17 18:54:48.744755 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:48.744702 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8" podStartSLOduration=14.744684264 podStartE2EDuration="14.744684264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:54:48.743839632 +0000 UTC m=+3484.153267232" watchObservedRunningTime="2026-07-17 18:54:48.744684264 +0000 UTC m=+3484.154111870" Jul 17 18:54:49.733379 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:49.733346 2572 generic.go:356] "Generic (PLEG): container finished" podID="b652d0ce-f8b5-4239-84f9-6d5227fb2b7b" containerID="bac4a7311cab335a478f7df8984cd139b6d4019eca908c01d258c8462f0099fa" exitCode=6 Jul 17 18:54:49.733819 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:49.733409 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8" event={"ID":"b652d0ce-f8b5-4239-84f9-6d5227fb2b7b","Type":"ContainerDied","Data":"bac4a7311cab335a478f7df8984cd139b6d4019eca908c01d258c8462f0099fa"} Jul 17 18:54:49.733819 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:49.733458 2572 scope.go:117] "RemoveContainer" containerID="0c868b6378f2c256127b949e6743e6ebc20203822660aca2678d7fb892dde572" Jul 17 18:54:49.733819 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:49.733722 2572 scope.go:117] "RemoveContainer" containerID="bac4a7311cab335a478f7df8984cd139b6d4019eca908c01d258c8462f0099fa" Jul 17 18:54:49.733979 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:54:49.733959 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-vttb8_odh-ai-gateway-infra(b652d0ce-f8b5-4239-84f9-6d5227fb2b7b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8" podUID="b652d0ce-f8b5-4239-84f9-6d5227fb2b7b" Jul 17 18:54:49.758842 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:49.758813 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8"] Jul 17 18:54:50.866837 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:50.866808 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8" Jul 17 18:54:50.899744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:50.899713 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b652d0ce-f8b5-4239-84f9-6d5227fb2b7b-maas-api-service-ca\") pod \"b652d0ce-f8b5-4239-84f9-6d5227fb2b7b\" (UID: \"b652d0ce-f8b5-4239-84f9-6d5227fb2b7b\") " Jul 17 18:54:50.900076 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:50.900052 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b652d0ce-f8b5-4239-84f9-6d5227fb2b7b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b652d0ce-f8b5-4239-84f9-6d5227fb2b7b" (UID: "b652d0ce-f8b5-4239-84f9-6d5227fb2b7b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:54:51.001136 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:51.001044 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b652d0ce-f8b5-4239-84f9-6d5227fb2b7b-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:54:51.744782 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:51.744739 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8" event={"ID":"b652d0ce-f8b5-4239-84f9-6d5227fb2b7b","Type":"ContainerDied","Data":"017e6a8ce19ef9b3684f5ed24e057b3c772c70bc968761044ba969fc437bc612"} Jul 17 18:54:51.745055 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:51.744814 2572 scope.go:117] "RemoveContainer" containerID="bac4a7311cab335a478f7df8984cd139b6d4019eca908c01d258c8462f0099fa" Jul 17 18:54:51.745055 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:51.744753 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8" Jul 17 18:54:51.760647 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:51.760623 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8"] Jul 17 18:54:51.766142 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:51.766120 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-vttb8"] Jul 17 18:54:53.224271 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:53.224238 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b652d0ce-f8b5-4239-84f9-6d5227fb2b7b" path="/var/lib/kubelet/pods/b652d0ce-f8b5-4239-84f9-6d5227fb2b7b/volumes" Jul 17 18:54:55.911545 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.911511 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4"] Jul 17 18:54:55.911924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.911791 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbbccdb2-974b-4870-8d62-b91e2d1d77f7" containerName="revoke-keys" Jul 17 18:54:55.911924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.911802 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbbccdb2-974b-4870-8d62-b91e2d1d77f7" containerName="revoke-keys" Jul 17 18:54:55.911924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.911810 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b652d0ce-f8b5-4239-84f9-6d5227fb2b7b" containerName="revoke-keys" Jul 17 18:54:55.911924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.911815 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b652d0ce-f8b5-4239-84f9-6d5227fb2b7b" containerName="revoke-keys" Jul 17 18:54:55.911924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.911827 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbbccdb2-974b-4870-8d62-b91e2d1d77f7" containerName="revoke-keys" Jul 17 18:54:55.911924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.911832 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbbccdb2-974b-4870-8d62-b91e2d1d77f7" containerName="revoke-keys" Jul 17 18:54:55.911924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.911845 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd861277-9b86-4ed7-b054-49c9e85da935" containerName="revoke-keys" Jul 17 18:54:55.911924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.911850 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd861277-9b86-4ed7-b054-49c9e85da935" containerName="revoke-keys" Jul 17 18:54:55.911924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.911856 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b652d0ce-f8b5-4239-84f9-6d5227fb2b7b" containerName="revoke-keys" Jul 17 18:54:55.911924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.911861 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b652d0ce-f8b5-4239-84f9-6d5227fb2b7b" containerName="revoke-keys" Jul 17 18:54:55.911924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.911908 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbbccdb2-974b-4870-8d62-b91e2d1d77f7" containerName="revoke-keys" Jul 17 18:54:55.911924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.911915 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b652d0ce-f8b5-4239-84f9-6d5227fb2b7b" containerName="revoke-keys" Jul 17 18:54:55.911924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.911921 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbbccdb2-974b-4870-8d62-b91e2d1d77f7" containerName="revoke-keys" Jul 17 18:54:55.911924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.911927 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b652d0ce-f8b5-4239-84f9-6d5227fb2b7b" containerName="revoke-keys" Jul 17 18:54:55.912385 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.911934 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd861277-9b86-4ed7-b054-49c9e85da935" containerName="revoke-keys" Jul 17 18:54:55.916168 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.916152 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4" Jul 17 18:54:55.918713 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.918690 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:54:55.918833 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.918808 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:54:55.923441 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.922896 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4"] Jul 17 18:54:55.942137 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:55.942109 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4\" (UID: \"779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4" Jul 17 18:54:56.043527 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:56.043484 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4\" (UID: \"779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4" Jul 17 18:54:56.044146 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:56.044126 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4\" (UID: \"779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4" Jul 17 18:54:56.226947 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:56.226859 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4" Jul 17 18:54:56.354657 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:56.354625 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4"] Jul 17 18:54:56.357208 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:54:56.357176 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779cb7fc_ad3e_4b7a_8e53_ec4c6d2ebd4f.slice/crio-36e7128aa50259fb22a6dcb7b00a45410c903a5fa623034ec7474cf56d683729 WatchSource:0}: Error finding container 36e7128aa50259fb22a6dcb7b00a45410c903a5fa623034ec7474cf56d683729: Status 404 returned error can't find the container with id 36e7128aa50259fb22a6dcb7b00a45410c903a5fa623034ec7474cf56d683729 Jul 17 18:54:56.768376 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:56.768336 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4" event={"ID":"779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f","Type":"ContainerStarted","Data":"2b8f2299f1e9ef1539abef2149cfc8864b06ddf388f8fb5a2104b2b0eeefc6fe"} Jul 17 18:54:56.768561 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:56.768396 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4" event={"ID":"779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f","Type":"ContainerStarted","Data":"36e7128aa50259fb22a6dcb7b00a45410c903a5fa623034ec7474cf56d683729"} Jul 17 18:54:56.783694 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:56.783643 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4" podStartSLOduration=1.7836221700000001 podStartE2EDuration="1.78362217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:54:56.782943989 +0000 UTC m=+3492.192371595" watchObservedRunningTime="2026-07-17 18:54:56.78362217 +0000 UTC m=+3492.193049775" Jul 17 18:54:57.773905 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:57.773870 2572 generic.go:356] "Generic (PLEG): container finished" podID="779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f" containerID="2b8f2299f1e9ef1539abef2149cfc8864b06ddf388f8fb5a2104b2b0eeefc6fe" exitCode=6 Jul 17 18:54:57.774329 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:57.773954 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4" event={"ID":"779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f","Type":"ContainerDied","Data":"2b8f2299f1e9ef1539abef2149cfc8864b06ddf388f8fb5a2104b2b0eeefc6fe"} Jul 17 18:54:57.774329 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:57.774130 2572 scope.go:117] "RemoveContainer" containerID="2b8f2299f1e9ef1539abef2149cfc8864b06ddf388f8fb5a2104b2b0eeefc6fe" Jul 17 18:54:58.038580 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.038560 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc"] Jul 17 18:54:58.038893 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.038879 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbbccdb2-974b-4870-8d62-b91e2d1d77f7" containerName="revoke-keys" Jul 17 18:54:58.038971 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.038897 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbbccdb2-974b-4870-8d62-b91e2d1d77f7" containerName="revoke-keys" Jul 17 18:54:58.038971 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.038908 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b652d0ce-f8b5-4239-84f9-6d5227fb2b7b" containerName="revoke-keys" Jul 17 18:54:58.038971 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.038918 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b652d0ce-f8b5-4239-84f9-6d5227fb2b7b" containerName="revoke-keys" Jul 17 18:54:58.039165 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.039035 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbbccdb2-974b-4870-8d62-b91e2d1d77f7" containerName="revoke-keys" Jul 17 18:54:58.039165 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.039049 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b652d0ce-f8b5-4239-84f9-6d5227fb2b7b" containerName="revoke-keys" Jul 17 18:54:58.041957 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.041936 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc" Jul 17 18:54:58.050582 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.050554 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc"] Jul 17 18:54:58.060289 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.060262 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/153c2c79-b5b7-43ff-969c-c44ab12ea4a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc\" (UID: \"153c2c79-b5b7-43ff-969c-c44ab12ea4a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc" Jul 17 18:54:58.161427 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.161392 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/153c2c79-b5b7-43ff-969c-c44ab12ea4a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc\" (UID: \"153c2c79-b5b7-43ff-969c-c44ab12ea4a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc" Jul 17 18:54:58.162044 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.161998 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/153c2c79-b5b7-43ff-969c-c44ab12ea4a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc\" (UID: \"153c2c79-b5b7-43ff-969c-c44ab12ea4a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc" Jul 17 18:54:58.354588 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.354559 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc" Jul 17 18:54:58.482451 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.482419 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc"] Jul 17 18:54:58.485187 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:54:58.485157 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153c2c79_b5b7_43ff_969c_c44ab12ea4a0.slice/crio-189251b652fc9df4a079fd9ad55006782d325ace3a4b826fa7c0316a184b04e4 WatchSource:0}: Error finding container 189251b652fc9df4a079fd9ad55006782d325ace3a4b826fa7c0316a184b04e4: Status 404 returned error can't find the container with id 189251b652fc9df4a079fd9ad55006782d325ace3a4b826fa7c0316a184b04e4 Jul 17 18:54:58.781023 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.780985 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc" event={"ID":"153c2c79-b5b7-43ff-969c-c44ab12ea4a0","Type":"ContainerStarted","Data":"189251b652fc9df4a079fd9ad55006782d325ace3a4b826fa7c0316a184b04e4"} Jul 17 18:54:58.782713 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.782688 2572 generic.go:356] "Generic (PLEG): container finished" podID="779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f" containerID="729c95d1f63cf0bacf9561609c8ae55f82a2ea3c07c54059dbe5ec4e4970209f" exitCode=6 Jul 17 18:54:58.782828 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.782778 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4" event={"ID":"779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f","Type":"ContainerDied","Data":"729c95d1f63cf0bacf9561609c8ae55f82a2ea3c07c54059dbe5ec4e4970209f"} Jul 17 18:54:58.782869 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.782833 2572 scope.go:117] "RemoveContainer" containerID="2b8f2299f1e9ef1539abef2149cfc8864b06ddf388f8fb5a2104b2b0eeefc6fe" Jul 17 18:54:58.783054 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:58.783039 2572 scope.go:117] "RemoveContainer" containerID="729c95d1f63cf0bacf9561609c8ae55f82a2ea3c07c54059dbe5ec4e4970209f" Jul 17 18:54:58.783299 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:54:58.783277 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4_odh-ai-gateway-infra(779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4" podUID="779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f" Jul 17 18:54:59.789287 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:59.789258 2572 generic.go:356] "Generic (PLEG): container finished" podID="153c2c79-b5b7-43ff-969c-c44ab12ea4a0" containerID="14d269c91278e15d45dcc72d37d215c6ee7f3b030941243c1d41fb6fc60a6e62" exitCode=6 Jul 17 18:54:59.789734 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:59.789345 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc" event={"ID":"153c2c79-b5b7-43ff-969c-c44ab12ea4a0","Type":"ContainerDied","Data":"14d269c91278e15d45dcc72d37d215c6ee7f3b030941243c1d41fb6fc60a6e62"} Jul 17 18:54:59.789734 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:59.789531 2572 scope.go:117] "RemoveContainer" containerID="14d269c91278e15d45dcc72d37d215c6ee7f3b030941243c1d41fb6fc60a6e62" Jul 17 18:54:59.791079 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:54:59.791063 2572 scope.go:117] "RemoveContainer" containerID="729c95d1f63cf0bacf9561609c8ae55f82a2ea3c07c54059dbe5ec4e4970209f" Jul 17 18:54:59.791266 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:54:59.791248 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4_odh-ai-gateway-infra(779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4" podUID="779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f" Jul 17 18:55:00.802351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:00.802305 2572 generic.go:356] "Generic (PLEG): container finished" podID="153c2c79-b5b7-43ff-969c-c44ab12ea4a0" containerID="ed6130f5e97321754d261cd909b3f8d47a8cbf116ed924bc07f010f567af5bd2" exitCode=6 Jul 17 18:55:00.802802 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:00.802376 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc" event={"ID":"153c2c79-b5b7-43ff-969c-c44ab12ea4a0","Type":"ContainerDied","Data":"ed6130f5e97321754d261cd909b3f8d47a8cbf116ed924bc07f010f567af5bd2"} Jul 17 18:55:00.802802 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:00.802455 2572 scope.go:117] "RemoveContainer" containerID="14d269c91278e15d45dcc72d37d215c6ee7f3b030941243c1d41fb6fc60a6e62" Jul 17 18:55:00.802802 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:00.802712 2572 scope.go:117] "RemoveContainer" containerID="ed6130f5e97321754d261cd909b3f8d47a8cbf116ed924bc07f010f567af5bd2" Jul 17 18:55:00.802923 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:55:00.802891 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc_odh-ai-gateway-infra(153c2c79-b5b7-43ff-969c-c44ab12ea4a0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc" podUID="153c2c79-b5b7-43ff-969c-c44ab12ea4a0" Jul 17 18:55:01.808719 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:01.808692 2572 scope.go:117] "RemoveContainer" containerID="ed6130f5e97321754d261cd909b3f8d47a8cbf116ed924bc07f010f567af5bd2" Jul 17 18:55:01.809126 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:55:01.808853 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc_odh-ai-gateway-infra(153c2c79-b5b7-43ff-969c-c44ab12ea4a0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc" podUID="153c2c79-b5b7-43ff-969c-c44ab12ea4a0" Jul 17 18:55:13.220191 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:13.220114 2572 scope.go:117] "RemoveContainer" containerID="ed6130f5e97321754d261cd909b3f8d47a8cbf116ed924bc07f010f567af5bd2" Jul 17 18:55:13.861989 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:13.861961 2572 generic.go:356] "Generic (PLEG): container finished" podID="153c2c79-b5b7-43ff-969c-c44ab12ea4a0" containerID="3df548a0e19136f5bb07bbdcd54dc295ed3308dd003d5d20644802aa207d99f4" exitCode=6 Jul 17 18:55:13.862107 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:13.862029 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc" event={"ID":"153c2c79-b5b7-43ff-969c-c44ab12ea4a0","Type":"ContainerDied","Data":"3df548a0e19136f5bb07bbdcd54dc295ed3308dd003d5d20644802aa207d99f4"} Jul 17 18:55:13.862146 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:13.862119 2572 scope.go:117] "RemoveContainer" containerID="ed6130f5e97321754d261cd909b3f8d47a8cbf116ed924bc07f010f567af5bd2" Jul 17 18:55:13.862443 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:13.862422 2572 scope.go:117] "RemoveContainer" containerID="3df548a0e19136f5bb07bbdcd54dc295ed3308dd003d5d20644802aa207d99f4" Jul 17 18:55:13.862645 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:55:13.862625 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc_odh-ai-gateway-infra(153c2c79-b5b7-43ff-969c-c44ab12ea4a0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc" podUID="153c2c79-b5b7-43ff-969c-c44ab12ea4a0" Jul 17 18:55:14.219840 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:14.219771 2572 scope.go:117] "RemoveContainer" containerID="729c95d1f63cf0bacf9561609c8ae55f82a2ea3c07c54059dbe5ec4e4970209f" Jul 17 18:55:14.869683 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:14.869659 2572 generic.go:356] "Generic (PLEG): container finished" podID="779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f" containerID="f8576599ea26f7eeea451084abdf3ac32e13a19d373c165d029335179e6324ba" exitCode=6 Jul 17 18:55:14.870039 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:14.869712 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4" event={"ID":"779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f","Type":"ContainerDied","Data":"f8576599ea26f7eeea451084abdf3ac32e13a19d373c165d029335179e6324ba"} Jul 17 18:55:14.870039 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:14.869761 2572 scope.go:117] "RemoveContainer" containerID="729c95d1f63cf0bacf9561609c8ae55f82a2ea3c07c54059dbe5ec4e4970209f" Jul 17 18:55:14.870135 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:14.870076 2572 scope.go:117] "RemoveContainer" containerID="f8576599ea26f7eeea451084abdf3ac32e13a19d373c165d029335179e6324ba" Jul 17 18:55:14.870329 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:55:14.870310 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4_odh-ai-gateway-infra(779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4" podUID="779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f" Jul 17 18:55:14.890084 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:14.890057 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc"] Jul 17 18:55:15.022203 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:15.022178 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc" Jul 17 18:55:15.111302 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:15.111229 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/153c2c79-b5b7-43ff-969c-c44ab12ea4a0-maas-api-service-ca\") pod \"153c2c79-b5b7-43ff-969c-c44ab12ea4a0\" (UID: \"153c2c79-b5b7-43ff-969c-c44ab12ea4a0\") " Jul 17 18:55:15.111586 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:15.111567 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153c2c79-b5b7-43ff-969c-c44ab12ea4a0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "153c2c79-b5b7-43ff-969c-c44ab12ea4a0" (UID: "153c2c79-b5b7-43ff-969c-c44ab12ea4a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:55:15.211868 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:15.211842 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/153c2c79-b5b7-43ff-969c-c44ab12ea4a0-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:55:15.875351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:15.875327 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc" Jul 17 18:55:15.875351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:15.875338 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc" event={"ID":"153c2c79-b5b7-43ff-969c-c44ab12ea4a0","Type":"ContainerDied","Data":"189251b652fc9df4a079fd9ad55006782d325ace3a4b826fa7c0316a184b04e4"} Jul 17 18:55:15.875924 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:15.875396 2572 scope.go:117] "RemoveContainer" containerID="3df548a0e19136f5bb07bbdcd54dc295ed3308dd003d5d20644802aa207d99f4" Jul 17 18:55:15.891156 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:15.891135 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc"] Jul 17 18:55:15.895576 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:15.895557 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4"] Jul 17 18:55:15.899344 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:15.899323 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-lbhqc"] Jul 17 18:55:16.021128 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:16.021104 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4" Jul 17 18:55:16.119056 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:16.119023 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f-maas-api-service-ca\") pod \"779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f\" (UID: \"779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f\") " Jul 17 18:55:16.119381 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:16.119358 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f" (UID: "779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:55:16.220091 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:16.220002 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:55:16.883652 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:16.883624 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4" Jul 17 18:55:16.884117 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:16.883621 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4" event={"ID":"779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f","Type":"ContainerDied","Data":"36e7128aa50259fb22a6dcb7b00a45410c903a5fa623034ec7474cf56d683729"} Jul 17 18:55:16.884117 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:16.883770 2572 scope.go:117] "RemoveContainer" containerID="f8576599ea26f7eeea451084abdf3ac32e13a19d373c165d029335179e6324ba" Jul 17 18:55:16.907206 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:16.907180 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4"] Jul 17 18:55:16.910643 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:16.910617 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-56mg4"] Jul 17 18:55:17.224783 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:17.224700 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="153c2c79-b5b7-43ff-969c-c44ab12ea4a0" path="/var/lib/kubelet/pods/153c2c79-b5b7-43ff-969c-c44ab12ea4a0/volumes" Jul 17 18:55:17.225060 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:17.225046 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f" path="/var/lib/kubelet/pods/779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f/volumes" Jul 17 18:55:21.529266 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.529235 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs"] Jul 17 18:55:21.529649 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.529544 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f" containerName="revoke-keys" Jul 17 18:55:21.529649 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.529556 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f" containerName="revoke-keys" Jul 17 18:55:21.529649 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.529575 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="153c2c79-b5b7-43ff-969c-c44ab12ea4a0" containerName="revoke-keys" Jul 17 18:55:21.529649 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.529583 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="153c2c79-b5b7-43ff-969c-c44ab12ea4a0" containerName="revoke-keys" Jul 17 18:55:21.529649 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.529598 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="153c2c79-b5b7-43ff-969c-c44ab12ea4a0" containerName="revoke-keys" Jul 17 18:55:21.529649 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.529604 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="153c2c79-b5b7-43ff-969c-c44ab12ea4a0" containerName="revoke-keys" Jul 17 18:55:21.529649 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.529611 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f" containerName="revoke-keys" Jul 17 18:55:21.529649 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.529616 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f" containerName="revoke-keys" Jul 17 18:55:21.529649 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.529625 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="153c2c79-b5b7-43ff-969c-c44ab12ea4a0" containerName="revoke-keys" Jul 17 18:55:21.529649 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.529630 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="153c2c79-b5b7-43ff-969c-c44ab12ea4a0" containerName="revoke-keys" Jul 17 18:55:21.529965 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.529680 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f" containerName="revoke-keys" Jul 17 18:55:21.529965 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.529689 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="153c2c79-b5b7-43ff-969c-c44ab12ea4a0" containerName="revoke-keys" Jul 17 18:55:21.529965 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.529697 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="153c2c79-b5b7-43ff-969c-c44ab12ea4a0" containerName="revoke-keys" Jul 17 18:55:21.529965 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.529702 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f" containerName="revoke-keys" Jul 17 18:55:21.529965 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.529709 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f" containerName="revoke-keys" Jul 17 18:55:21.534390 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.534372 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs" Jul 17 18:55:21.537208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.537176 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:55:21.537208 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.537204 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:55:21.540605 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.540581 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs"] Jul 17 18:55:21.667832 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.667797 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d731ea82-0050-4e31-87e3-cfb5aeea6f13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs\" (UID: \"d731ea82-0050-4e31-87e3-cfb5aeea6f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs" Jul 17 18:55:21.768246 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.768216 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d731ea82-0050-4e31-87e3-cfb5aeea6f13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs\" (UID: \"d731ea82-0050-4e31-87e3-cfb5aeea6f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs" Jul 17 18:55:21.768790 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.768772 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d731ea82-0050-4e31-87e3-cfb5aeea6f13-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs\" (UID: \"d731ea82-0050-4e31-87e3-cfb5aeea6f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs" Jul 17 18:55:21.845527 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.845448 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs" Jul 17 18:55:21.971190 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:21.971163 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs"] Jul 17 18:55:21.973077 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:55:21.973049 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd731ea82_0050_4e31_87e3_cfb5aeea6f13.slice/crio-17017a4f36f1331729e23cb7cc688bb90431c8797cecd420bbdf81a8fdbd8753 WatchSource:0}: Error finding container 17017a4f36f1331729e23cb7cc688bb90431c8797cecd420bbdf81a8fdbd8753: Status 404 returned error can't find the container with id 17017a4f36f1331729e23cb7cc688bb90431c8797cecd420bbdf81a8fdbd8753 Jul 17 18:55:22.912452 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:22.912369 2572 generic.go:356] "Generic (PLEG): container finished" podID="d731ea82-0050-4e31-87e3-cfb5aeea6f13" containerID="d0c2b7f01701cbf2211a86429492cc316738270f00bf68501c5e2aef5304334f" exitCode=6 Jul 17 18:55:22.912923 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:22.912454 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs" event={"ID":"d731ea82-0050-4e31-87e3-cfb5aeea6f13","Type":"ContainerDied","Data":"d0c2b7f01701cbf2211a86429492cc316738270f00bf68501c5e2aef5304334f"} Jul 17 18:55:22.912923 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:22.912505 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs" event={"ID":"d731ea82-0050-4e31-87e3-cfb5aeea6f13","Type":"ContainerStarted","Data":"17017a4f36f1331729e23cb7cc688bb90431c8797cecd420bbdf81a8fdbd8753"} Jul 17 18:55:22.912923 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:22.912623 2572 scope.go:117] "RemoveContainer" containerID="d0c2b7f01701cbf2211a86429492cc316738270f00bf68501c5e2aef5304334f" Jul 17 18:55:23.918450 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:23.918357 2572 generic.go:356] "Generic (PLEG): container finished" podID="d731ea82-0050-4e31-87e3-cfb5aeea6f13" containerID="d55d6741b97b7f627dd8b3b02a14fce69b1452954a916790c865a2dc5e7713a7" exitCode=6 Jul 17 18:55:23.918450 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:23.918424 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs" event={"ID":"d731ea82-0050-4e31-87e3-cfb5aeea6f13","Type":"ContainerDied","Data":"d55d6741b97b7f627dd8b3b02a14fce69b1452954a916790c865a2dc5e7713a7"} Jul 17 18:55:23.918950 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:23.918486 2572 scope.go:117] "RemoveContainer" containerID="d0c2b7f01701cbf2211a86429492cc316738270f00bf68501c5e2aef5304334f" Jul 17 18:55:23.918950 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:23.918671 2572 scope.go:117] "RemoveContainer" containerID="d55d6741b97b7f627dd8b3b02a14fce69b1452954a916790c865a2dc5e7713a7" Jul 17 18:55:23.918950 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:55:23.918915 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs_odh-ai-gateway-infra(d731ea82-0050-4e31-87e3-cfb5aeea6f13)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs" podUID="d731ea82-0050-4e31-87e3-cfb5aeea6f13" Jul 17 18:55:24.087487 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:24.087452 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn"] Jul 17 18:55:24.087761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:24.087749 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f" containerName="revoke-keys" Jul 17 18:55:24.087805 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:24.087763 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="779cb7fc-ad3e-4b7a-8e53-ec4c6d2ebd4f" containerName="revoke-keys" Jul 17 18:55:24.087842 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:24.087817 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="153c2c79-b5b7-43ff-969c-c44ab12ea4a0" containerName="revoke-keys" Jul 17 18:55:24.091829 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:24.091813 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn" Jul 17 18:55:24.098229 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:24.098201 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn"] Jul 17 18:55:24.188394 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:24.188314 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4758290-591d-491c-b4fb-a3c69f244f56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn\" (UID: \"c4758290-591d-491c-b4fb-a3c69f244f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn" Jul 17 18:55:24.289037 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:24.288969 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4758290-591d-491c-b4fb-a3c69f244f56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn\" (UID: \"c4758290-591d-491c-b4fb-a3c69f244f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn" Jul 17 18:55:24.289614 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:24.289595 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4758290-591d-491c-b4fb-a3c69f244f56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn\" (UID: \"c4758290-591d-491c-b4fb-a3c69f244f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn" Jul 17 18:55:24.403672 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:24.403643 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn" Jul 17 18:55:24.526810 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:24.526769 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn"] Jul 17 18:55:24.529425 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:55:24.529396 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4758290_591d_491c_b4fb_a3c69f244f56.slice/crio-bb77fa8e23d07b33b72ba1d902aeab27c64991fa68566dd719bb6e840a88b5c1 WatchSource:0}: Error finding container bb77fa8e23d07b33b72ba1d902aeab27c64991fa68566dd719bb6e840a88b5c1: Status 404 returned error can't find the container with id bb77fa8e23d07b33b72ba1d902aeab27c64991fa68566dd719bb6e840a88b5c1 Jul 17 18:55:24.924471 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:24.924443 2572 scope.go:117] "RemoveContainer" containerID="d55d6741b97b7f627dd8b3b02a14fce69b1452954a916790c865a2dc5e7713a7" Jul 17 18:55:24.924896 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:55:24.924648 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs_odh-ai-gateway-infra(d731ea82-0050-4e31-87e3-cfb5aeea6f13)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs" podUID="d731ea82-0050-4e31-87e3-cfb5aeea6f13" Jul 17 18:55:24.925654 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:24.925633 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn" event={"ID":"c4758290-591d-491c-b4fb-a3c69f244f56","Type":"ContainerStarted","Data":"9dc992495d11cbb5618b592e52aa72afb8312a280a27652d10d219f87824f87e"} Jul 17 18:55:24.925749 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:24.925673 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn" event={"ID":"c4758290-591d-491c-b4fb-a3c69f244f56","Type":"ContainerStarted","Data":"bb77fa8e23d07b33b72ba1d902aeab27c64991fa68566dd719bb6e840a88b5c1"} Jul 17 18:55:24.973474 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:24.973430 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn" podStartSLOduration=0.973416836 podStartE2EDuration="973.416836ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:24.970938556 +0000 UTC m=+3520.380366173" watchObservedRunningTime="2026-07-17 18:55:24.973416836 +0000 UTC m=+3520.382844441" Jul 17 18:55:25.931427 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:25.931397 2572 generic.go:356] "Generic (PLEG): container finished" podID="c4758290-591d-491c-b4fb-a3c69f244f56" containerID="9dc992495d11cbb5618b592e52aa72afb8312a280a27652d10d219f87824f87e" exitCode=6 Jul 17 18:55:25.931825 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:25.931481 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn" event={"ID":"c4758290-591d-491c-b4fb-a3c69f244f56","Type":"ContainerDied","Data":"9dc992495d11cbb5618b592e52aa72afb8312a280a27652d10d219f87824f87e"} Jul 17 18:55:25.931825 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:25.931662 2572 scope.go:117] "RemoveContainer" containerID="9dc992495d11cbb5618b592e52aa72afb8312a280a27652d10d219f87824f87e" Jul 17 18:55:26.937730 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:26.937639 2572 generic.go:356] "Generic (PLEG): container finished" podID="c4758290-591d-491c-b4fb-a3c69f244f56" containerID="2c004db9d2925cb7724482509f85ac5cf00c85727e396469ca5dd9e0249686da" exitCode=6 Jul 17 18:55:26.937730 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:26.937677 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn" event={"ID":"c4758290-591d-491c-b4fb-a3c69f244f56","Type":"ContainerDied","Data":"2c004db9d2925cb7724482509f85ac5cf00c85727e396469ca5dd9e0249686da"} Jul 17 18:55:26.937730 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:26.937729 2572 scope.go:117] "RemoveContainer" containerID="9dc992495d11cbb5618b592e52aa72afb8312a280a27652d10d219f87824f87e" Jul 17 18:55:26.938262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:26.937915 2572 scope.go:117] "RemoveContainer" containerID="2c004db9d2925cb7724482509f85ac5cf00c85727e396469ca5dd9e0249686da" Jul 17 18:55:26.938262 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:55:26.938128 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn_odh-ai-gateway-infra(c4758290-591d-491c-b4fb-a3c69f244f56)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn" podUID="c4758290-591d-491c-b4fb-a3c69f244f56" Jul 17 18:55:27.944563 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:27.944528 2572 scope.go:117] "RemoveContainer" containerID="2c004db9d2925cb7724482509f85ac5cf00c85727e396469ca5dd9e0249686da" Jul 17 18:55:27.944933 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:55:27.944745 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn_odh-ai-gateway-infra(c4758290-591d-491c-b4fb-a3c69f244f56)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn" podUID="c4758290-591d-491c-b4fb-a3c69f244f56" Jul 17 18:55:37.220045 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:37.219996 2572 scope.go:117] "RemoveContainer" containerID="d55d6741b97b7f627dd8b3b02a14fce69b1452954a916790c865a2dc5e7713a7" Jul 17 18:55:37.993156 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:37.993124 2572 generic.go:356] "Generic (PLEG): container finished" podID="d731ea82-0050-4e31-87e3-cfb5aeea6f13" containerID="b683cf2b774c503ea4e729e00cc036c09b5e83d74442b5d725a32d8b44d0062c" exitCode=6 Jul 17 18:55:37.993322 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:37.993191 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs" event={"ID":"d731ea82-0050-4e31-87e3-cfb5aeea6f13","Type":"ContainerDied","Data":"b683cf2b774c503ea4e729e00cc036c09b5e83d74442b5d725a32d8b44d0062c"} Jul 17 18:55:37.993322 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:37.993253 2572 scope.go:117] "RemoveContainer" containerID="d55d6741b97b7f627dd8b3b02a14fce69b1452954a916790c865a2dc5e7713a7" Jul 17 18:55:37.993549 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:37.993535 2572 scope.go:117] "RemoveContainer" containerID="b683cf2b774c503ea4e729e00cc036c09b5e83d74442b5d725a32d8b44d0062c" Jul 17 18:55:37.993753 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:55:37.993728 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs_odh-ai-gateway-infra(d731ea82-0050-4e31-87e3-cfb5aeea6f13)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs" podUID="d731ea82-0050-4e31-87e3-cfb5aeea6f13" Jul 17 18:55:39.019082 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:39.019046 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs"] Jul 17 18:55:39.141699 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:39.141672 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs" Jul 17 18:55:39.215065 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:39.215034 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d731ea82-0050-4e31-87e3-cfb5aeea6f13-maas-api-service-ca\") pod \"d731ea82-0050-4e31-87e3-cfb5aeea6f13\" (UID: \"d731ea82-0050-4e31-87e3-cfb5aeea6f13\") " Jul 17 18:55:39.215374 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:39.215350 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d731ea82-0050-4e31-87e3-cfb5aeea6f13-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d731ea82-0050-4e31-87e3-cfb5aeea6f13" (UID: "d731ea82-0050-4e31-87e3-cfb5aeea6f13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:55:39.315691 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:39.315665 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d731ea82-0050-4e31-87e3-cfb5aeea6f13-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:55:40.005103 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:40.005077 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs" Jul 17 18:55:40.005297 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:40.005124 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs" event={"ID":"d731ea82-0050-4e31-87e3-cfb5aeea6f13","Type":"ContainerDied","Data":"17017a4f36f1331729e23cb7cc688bb90431c8797cecd420bbdf81a8fdbd8753"} Jul 17 18:55:40.005297 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:40.005181 2572 scope.go:117] "RemoveContainer" containerID="b683cf2b774c503ea4e729e00cc036c09b5e83d74442b5d725a32d8b44d0062c" Jul 17 18:55:40.021849 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:40.021818 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs"] Jul 17 18:55:40.031364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:40.028755 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-25hfs"] Jul 17 18:55:40.219698 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:40.219664 2572 scope.go:117] "RemoveContainer" containerID="2c004db9d2925cb7724482509f85ac5cf00c85727e396469ca5dd9e0249686da" Jul 17 18:55:41.012395 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:41.012363 2572 generic.go:356] "Generic (PLEG): container finished" podID="c4758290-591d-491c-b4fb-a3c69f244f56" containerID="e45416239c0b7136faced2072b96d7c8e05a5bd39375d1ee345c563d15c1e537" exitCode=6 Jul 17 18:55:41.012395 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:41.012398 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn" event={"ID":"c4758290-591d-491c-b4fb-a3c69f244f56","Type":"ContainerDied","Data":"e45416239c0b7136faced2072b96d7c8e05a5bd39375d1ee345c563d15c1e537"} Jul 17 18:55:41.012611 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:41.012452 2572 scope.go:117] "RemoveContainer" containerID="2c004db9d2925cb7724482509f85ac5cf00c85727e396469ca5dd9e0249686da" Jul 17 18:55:41.012739 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:41.012713 2572 scope.go:117] "RemoveContainer" containerID="e45416239c0b7136faced2072b96d7c8e05a5bd39375d1ee345c563d15c1e537" Jul 17 18:55:41.012941 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:55:41.012918 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn_odh-ai-gateway-infra(c4758290-591d-491c-b4fb-a3c69f244f56)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn" podUID="c4758290-591d-491c-b4fb-a3c69f244f56" Jul 17 18:55:41.226706 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:41.226626 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d731ea82-0050-4e31-87e3-cfb5aeea6f13" path="/var/lib/kubelet/pods/d731ea82-0050-4e31-87e3-cfb5aeea6f13/volumes" Jul 17 18:55:42.036505 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:42.036469 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn"] Jul 17 18:55:42.166196 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:42.166172 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn" Jul 17 18:55:42.245649 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:42.245617 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4758290-591d-491c-b4fb-a3c69f244f56-maas-api-service-ca\") pod \"c4758290-591d-491c-b4fb-a3c69f244f56\" (UID: \"c4758290-591d-491c-b4fb-a3c69f244f56\") " Jul 17 18:55:42.246061 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:42.245981 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4758290-591d-491c-b4fb-a3c69f244f56-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4758290-591d-491c-b4fb-a3c69f244f56" (UID: "c4758290-591d-491c-b4fb-a3c69f244f56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:55:42.347082 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:42.347003 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4758290-591d-491c-b4fb-a3c69f244f56-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:55:43.024424 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:43.024396 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn" Jul 17 18:55:43.024588 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:43.024395 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn" event={"ID":"c4758290-591d-491c-b4fb-a3c69f244f56","Type":"ContainerDied","Data":"bb77fa8e23d07b33b72ba1d902aeab27c64991fa68566dd719bb6e840a88b5c1"} Jul 17 18:55:43.024588 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:43.024528 2572 scope.go:117] "RemoveContainer" containerID="e45416239c0b7136faced2072b96d7c8e05a5bd39375d1ee345c563d15c1e537" Jul 17 18:55:43.048147 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:43.048118 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn"] Jul 17 18:55:43.053236 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:43.053215 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rvxkn"] Jul 17 18:55:43.225568 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:43.225534 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4758290-591d-491c-b4fb-a3c69f244f56" path="/var/lib/kubelet/pods/c4758290-591d-491c-b4fb-a3c69f244f56/volumes" Jul 17 18:55:48.081154 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.081121 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7"] Jul 17 18:55:48.081550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.081396 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d731ea82-0050-4e31-87e3-cfb5aeea6f13" containerName="revoke-keys" Jul 17 18:55:48.081550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.081406 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d731ea82-0050-4e31-87e3-cfb5aeea6f13" containerName="revoke-keys" Jul 17 18:55:48.081550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.081416 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4758290-591d-491c-b4fb-a3c69f244f56" containerName="revoke-keys" Jul 17 18:55:48.081550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.081422 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4758290-591d-491c-b4fb-a3c69f244f56" containerName="revoke-keys" Jul 17 18:55:48.081550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.081431 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d731ea82-0050-4e31-87e3-cfb5aeea6f13" containerName="revoke-keys" Jul 17 18:55:48.081550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.081436 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d731ea82-0050-4e31-87e3-cfb5aeea6f13" containerName="revoke-keys" Jul 17 18:55:48.081550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.081447 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d731ea82-0050-4e31-87e3-cfb5aeea6f13" containerName="revoke-keys" Jul 17 18:55:48.081550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.081453 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d731ea82-0050-4e31-87e3-cfb5aeea6f13" containerName="revoke-keys" Jul 17 18:55:48.081550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.081459 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4758290-591d-491c-b4fb-a3c69f244f56" containerName="revoke-keys" Jul 17 18:55:48.081550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.081464 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4758290-591d-491c-b4fb-a3c69f244f56" containerName="revoke-keys" Jul 17 18:55:48.081550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.081524 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4758290-591d-491c-b4fb-a3c69f244f56" containerName="revoke-keys" Jul 17 18:55:48.081550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.081532 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4758290-591d-491c-b4fb-a3c69f244f56" containerName="revoke-keys" Jul 17 18:55:48.081550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.081541 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4758290-591d-491c-b4fb-a3c69f244f56" containerName="revoke-keys" Jul 17 18:55:48.081550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.081546 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d731ea82-0050-4e31-87e3-cfb5aeea6f13" containerName="revoke-keys" Jul 17 18:55:48.081550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.081552 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d731ea82-0050-4e31-87e3-cfb5aeea6f13" containerName="revoke-keys" Jul 17 18:55:48.081550 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.081557 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d731ea82-0050-4e31-87e3-cfb5aeea6f13" containerName="revoke-keys" Jul 17 18:55:48.086082 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.086066 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7" Jul 17 18:55:48.088773 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.088747 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:55:48.089066 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.089042 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:55:48.092477 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.092453 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7"] Jul 17 18:55:48.196847 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.196816 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06de94b3-4c22-4dec-9265-e77e606da446-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7\" (UID: \"06de94b3-4c22-4dec-9265-e77e606da446\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7" Jul 17 18:55:48.298164 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.298125 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06de94b3-4c22-4dec-9265-e77e606da446-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7\" (UID: \"06de94b3-4c22-4dec-9265-e77e606da446\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7" Jul 17 18:55:48.298764 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.298744 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06de94b3-4c22-4dec-9265-e77e606da446-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7\" (UID: \"06de94b3-4c22-4dec-9265-e77e606da446\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7" Jul 17 18:55:48.398167 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.398100 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7" Jul 17 18:55:48.522491 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:48.522457 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7"] Jul 17 18:55:48.525519 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:55:48.525495 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06de94b3_4c22_4dec_9265_e77e606da446.slice/crio-1be9988354f7f6c8b9ca2f979dd9f7f6fb3ae97ccac36d57f8e72c12a97478ac WatchSource:0}: Error finding container 1be9988354f7f6c8b9ca2f979dd9f7f6fb3ae97ccac36d57f8e72c12a97478ac: Status 404 returned error can't find the container with id 1be9988354f7f6c8b9ca2f979dd9f7f6fb3ae97ccac36d57f8e72c12a97478ac Jul 17 18:55:49.052627 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:49.052592 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7" event={"ID":"06de94b3-4c22-4dec-9265-e77e606da446","Type":"ContainerStarted","Data":"cc020a79d95ae506c196b6d45bbfa6eab3f571044944ac17247721813d6eca3a"} Jul 17 18:55:49.052806 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:49.052650 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7" event={"ID":"06de94b3-4c22-4dec-9265-e77e606da446","Type":"ContainerStarted","Data":"1be9988354f7f6c8b9ca2f979dd9f7f6fb3ae97ccac36d57f8e72c12a97478ac"} Jul 17 18:55:49.068024 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:49.067966 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7" podStartSLOduration=1.067951536 podStartE2EDuration="1.067951536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:49.06620147 +0000 UTC m=+3544.475629066" watchObservedRunningTime="2026-07-17 18:55:49.067951536 +0000 UTC m=+3544.477379141" Jul 17 18:55:50.058385 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:50.058349 2572 generic.go:356] "Generic (PLEG): container finished" podID="06de94b3-4c22-4dec-9265-e77e606da446" containerID="cc020a79d95ae506c196b6d45bbfa6eab3f571044944ac17247721813d6eca3a" exitCode=6 Jul 17 18:55:50.058751 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:50.058433 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7" event={"ID":"06de94b3-4c22-4dec-9265-e77e606da446","Type":"ContainerDied","Data":"cc020a79d95ae506c196b6d45bbfa6eab3f571044944ac17247721813d6eca3a"} Jul 17 18:55:50.058751 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:50.058597 2572 scope.go:117] "RemoveContainer" containerID="cc020a79d95ae506c196b6d45bbfa6eab3f571044944ac17247721813d6eca3a" Jul 17 18:55:51.064339 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:51.064302 2572 generic.go:356] "Generic (PLEG): container finished" podID="06de94b3-4c22-4dec-9265-e77e606da446" containerID="13f173793034660957324d7b4373e881b9acfe2eeb94b52f9ffec7297c2f7847" exitCode=6 Jul 17 18:55:51.064794 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:51.064385 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7" event={"ID":"06de94b3-4c22-4dec-9265-e77e606da446","Type":"ContainerDied","Data":"13f173793034660957324d7b4373e881b9acfe2eeb94b52f9ffec7297c2f7847"} Jul 17 18:55:51.064794 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:51.064495 2572 scope.go:117] "RemoveContainer" containerID="cc020a79d95ae506c196b6d45bbfa6eab3f571044944ac17247721813d6eca3a" Jul 17 18:55:51.064794 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:51.064722 2572 scope.go:117] "RemoveContainer" containerID="13f173793034660957324d7b4373e881b9acfe2eeb94b52f9ffec7297c2f7847" Jul 17 18:55:51.064979 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:55:51.064935 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7_odh-ai-gateway-infra(06de94b3-4c22-4dec-9265-e77e606da446)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7" podUID="06de94b3-4c22-4dec-9265-e77e606da446" Jul 17 18:55:52.070420 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:52.070392 2572 scope.go:117] "RemoveContainer" containerID="13f173793034660957324d7b4373e881b9acfe2eeb94b52f9ffec7297c2f7847" Jul 17 18:55:52.070779 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:55:52.070561 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7_odh-ai-gateway-infra(06de94b3-4c22-4dec-9265-e77e606da446)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7" podUID="06de94b3-4c22-4dec-9265-e77e606da446" Jul 17 18:55:55.954832 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:55.954797 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh"] Jul 17 18:55:55.955328 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:55.955124 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4758290-591d-491c-b4fb-a3c69f244f56" containerName="revoke-keys" Jul 17 18:55:55.955328 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:55.955137 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4758290-591d-491c-b4fb-a3c69f244f56" containerName="revoke-keys" Jul 17 18:55:55.958959 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:55.958940 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh" Jul 17 18:55:55.963801 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:55.963775 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh"] Jul 17 18:55:56.064618 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:56.064576 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b05351a-2be2-4b43-a273-b6eddab39909-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh\" (UID: \"1b05351a-2be2-4b43-a273-b6eddab39909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh" Jul 17 18:55:56.165258 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:56.165227 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b05351a-2be2-4b43-a273-b6eddab39909-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh\" (UID: \"1b05351a-2be2-4b43-a273-b6eddab39909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh" Jul 17 18:55:56.165812 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:56.165794 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b05351a-2be2-4b43-a273-b6eddab39909-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh\" (UID: \"1b05351a-2be2-4b43-a273-b6eddab39909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh" Jul 17 18:55:56.270118 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:56.270037 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh" Jul 17 18:55:56.396182 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:56.396149 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh"] Jul 17 18:55:56.399838 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:55:56.399814 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b05351a_2be2_4b43_a273_b6eddab39909.slice/crio-d31fead1612eadd1f650926f4c39d3343826849a1bc22420a896b53e0bbf4904 WatchSource:0}: Error finding container d31fead1612eadd1f650926f4c39d3343826849a1bc22420a896b53e0bbf4904: Status 404 returned error can't find the container with id d31fead1612eadd1f650926f4c39d3343826849a1bc22420a896b53e0bbf4904 Jul 17 18:55:57.092789 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:57.092758 2572 generic.go:356] "Generic (PLEG): container finished" podID="1b05351a-2be2-4b43-a273-b6eddab39909" containerID="870162c03aaadf31f452494ab5ea08961adffc5ec73f45194ad46c631353f214" exitCode=6 Jul 17 18:55:57.093192 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:57.092842 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh" event={"ID":"1b05351a-2be2-4b43-a273-b6eddab39909","Type":"ContainerDied","Data":"870162c03aaadf31f452494ab5ea08961adffc5ec73f45194ad46c631353f214"} Jul 17 18:55:57.093192 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:57.092892 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh" event={"ID":"1b05351a-2be2-4b43-a273-b6eddab39909","Type":"ContainerStarted","Data":"d31fead1612eadd1f650926f4c39d3343826849a1bc22420a896b53e0bbf4904"} Jul 17 18:55:57.093192 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:57.093109 2572 scope.go:117] "RemoveContainer" containerID="870162c03aaadf31f452494ab5ea08961adffc5ec73f45194ad46c631353f214" Jul 17 18:55:58.099087 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:58.099047 2572 generic.go:356] "Generic (PLEG): container finished" podID="1b05351a-2be2-4b43-a273-b6eddab39909" containerID="09f09c09952da27ec279467902f5bbd09cc9de2c8d9518931c97f234b5b6ec16" exitCode=6 Jul 17 18:55:58.099488 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:58.099140 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh" event={"ID":"1b05351a-2be2-4b43-a273-b6eddab39909","Type":"ContainerDied","Data":"09f09c09952da27ec279467902f5bbd09cc9de2c8d9518931c97f234b5b6ec16"} Jul 17 18:55:58.099488 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:58.099232 2572 scope.go:117] "RemoveContainer" containerID="870162c03aaadf31f452494ab5ea08961adffc5ec73f45194ad46c631353f214" Jul 17 18:55:58.099488 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:58.099322 2572 scope.go:117] "RemoveContainer" containerID="09f09c09952da27ec279467902f5bbd09cc9de2c8d9518931c97f234b5b6ec16" Jul 17 18:55:58.099668 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:55:58.099526 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh_odh-ai-gateway-infra(1b05351a-2be2-4b43-a273-b6eddab39909)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh" podUID="1b05351a-2be2-4b43-a273-b6eddab39909" Jul 17 18:55:59.105563 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:55:59.105539 2572 scope.go:117] "RemoveContainer" containerID="09f09c09952da27ec279467902f5bbd09cc9de2c8d9518931c97f234b5b6ec16" Jul 17 18:55:59.105928 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:55:59.105728 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh_odh-ai-gateway-infra(1b05351a-2be2-4b43-a273-b6eddab39909)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh" podUID="1b05351a-2be2-4b43-a273-b6eddab39909" Jul 17 18:56:03.219778 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:03.219747 2572 scope.go:117] "RemoveContainer" containerID="13f173793034660957324d7b4373e881b9acfe2eeb94b52f9ffec7297c2f7847" Jul 17 18:56:04.129229 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:04.129138 2572 generic.go:356] "Generic (PLEG): container finished" podID="06de94b3-4c22-4dec-9265-e77e606da446" containerID="0846a43ee1c26b8dc23238ced2b330f3fb36d6284a514c0c4f949a36065c6794" exitCode=6 Jul 17 18:56:04.129229 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:04.129211 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7" event={"ID":"06de94b3-4c22-4dec-9265-e77e606da446","Type":"ContainerDied","Data":"0846a43ee1c26b8dc23238ced2b330f3fb36d6284a514c0c4f949a36065c6794"} Jul 17 18:56:04.129465 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:04.129272 2572 scope.go:117] "RemoveContainer" containerID="13f173793034660957324d7b4373e881b9acfe2eeb94b52f9ffec7297c2f7847" Jul 17 18:56:04.129610 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:04.129596 2572 scope.go:117] "RemoveContainer" containerID="0846a43ee1c26b8dc23238ced2b330f3fb36d6284a514c0c4f949a36065c6794" Jul 17 18:56:04.129844 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:56:04.129823 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7_odh-ai-gateway-infra(06de94b3-4c22-4dec-9265-e77e606da446)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7" podUID="06de94b3-4c22-4dec-9265-e77e606da446" Jul 17 18:56:05.156988 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:05.156953 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7"] Jul 17 18:56:05.286852 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:05.286826 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7" Jul 17 18:56:05.449669 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:05.449586 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06de94b3-4c22-4dec-9265-e77e606da446-maas-api-service-ca\") pod \"06de94b3-4c22-4dec-9265-e77e606da446\" (UID: \"06de94b3-4c22-4dec-9265-e77e606da446\") " Jul 17 18:56:05.449976 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:05.449954 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06de94b3-4c22-4dec-9265-e77e606da446-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "06de94b3-4c22-4dec-9265-e77e606da446" (UID: "06de94b3-4c22-4dec-9265-e77e606da446"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:56:05.550777 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:05.550746 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06de94b3-4c22-4dec-9265-e77e606da446-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:56:06.143460 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:06.143439 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7" Jul 17 18:56:06.143636 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:06.143433 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7" event={"ID":"06de94b3-4c22-4dec-9265-e77e606da446","Type":"ContainerDied","Data":"1be9988354f7f6c8b9ca2f979dd9f7f6fb3ae97ccac36d57f8e72c12a97478ac"} Jul 17 18:56:06.143636 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:06.143558 2572 scope.go:117] "RemoveContainer" containerID="0846a43ee1c26b8dc23238ced2b330f3fb36d6284a514c0c4f949a36065c6794" Jul 17 18:56:06.163849 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:06.163824 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7"] Jul 17 18:56:06.169822 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:06.169800 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-gsnt7"] Jul 17 18:56:07.224734 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:07.224701 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06de94b3-4c22-4dec-9265-e77e606da446" path="/var/lib/kubelet/pods/06de94b3-4c22-4dec-9265-e77e606da446/volumes" Jul 17 18:56:11.571150 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:11.571115 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc"] Jul 17 18:56:11.571630 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:11.571404 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06de94b3-4c22-4dec-9265-e77e606da446" containerName="revoke-keys" Jul 17 18:56:11.571630 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:11.571415 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="06de94b3-4c22-4dec-9265-e77e606da446" containerName="revoke-keys" Jul 17 18:56:11.571630 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:11.571428 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06de94b3-4c22-4dec-9265-e77e606da446" containerName="revoke-keys" Jul 17 18:56:11.571630 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:11.571433 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="06de94b3-4c22-4dec-9265-e77e606da446" containerName="revoke-keys" Jul 17 18:56:11.571630 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:11.571489 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="06de94b3-4c22-4dec-9265-e77e606da446" containerName="revoke-keys" Jul 17 18:56:11.571630 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:11.571499 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="06de94b3-4c22-4dec-9265-e77e606da446" containerName="revoke-keys" Jul 17 18:56:11.571630 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:11.571505 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="06de94b3-4c22-4dec-9265-e77e606da446" containerName="revoke-keys" Jul 17 18:56:11.575507 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:11.575490 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc" Jul 17 18:56:11.581676 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:11.581653 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc"] Jul 17 18:56:11.706272 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:11.706235 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c52746-88e6-4cd9-9732-701ef0386536-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc\" (UID: \"58c52746-88e6-4cd9-9732-701ef0386536\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc" Jul 17 18:56:11.807370 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:11.807330 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c52746-88e6-4cd9-9732-701ef0386536-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc\" (UID: \"58c52746-88e6-4cd9-9732-701ef0386536\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc" Jul 17 18:56:11.807941 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:11.807923 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c52746-88e6-4cd9-9732-701ef0386536-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc\" (UID: \"58c52746-88e6-4cd9-9732-701ef0386536\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc" Jul 17 18:56:11.887080 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:11.887023 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc" Jul 17 18:56:12.215112 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:12.215053 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc"] Jul 17 18:56:12.217258 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:56:12.217216 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c52746_88e6_4cd9_9732_701ef0386536.slice/crio-f3dc9dfb436f1eb5b85e6c690307d0e1b1f6e26a2a89447f9fcce38c4ee30bd3 WatchSource:0}: Error finding container f3dc9dfb436f1eb5b85e6c690307d0e1b1f6e26a2a89447f9fcce38c4ee30bd3: Status 404 returned error can't find the container with id f3dc9dfb436f1eb5b85e6c690307d0e1b1f6e26a2a89447f9fcce38c4ee30bd3 Jul 17 18:56:13.176266 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:13.176226 2572 generic.go:356] "Generic (PLEG): container finished" podID="58c52746-88e6-4cd9-9732-701ef0386536" containerID="5824bf1c7e53b790742832456e5a89cbe911986d943e62fdd407c1954079d271" exitCode=6 Jul 17 18:56:13.176738 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:13.176291 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc" event={"ID":"58c52746-88e6-4cd9-9732-701ef0386536","Type":"ContainerDied","Data":"5824bf1c7e53b790742832456e5a89cbe911986d943e62fdd407c1954079d271"} Jul 17 18:56:13.176738 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:13.176349 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc" event={"ID":"58c52746-88e6-4cd9-9732-701ef0386536","Type":"ContainerStarted","Data":"f3dc9dfb436f1eb5b85e6c690307d0e1b1f6e26a2a89447f9fcce38c4ee30bd3"} Jul 17 18:56:13.176738 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:13.176545 2572 scope.go:117] "RemoveContainer" containerID="5824bf1c7e53b790742832456e5a89cbe911986d943e62fdd407c1954079d271" Jul 17 18:56:13.220125 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:13.220099 2572 scope.go:117] "RemoveContainer" containerID="09f09c09952da27ec279467902f5bbd09cc9de2c8d9518931c97f234b5b6ec16" Jul 17 18:56:14.125821 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:14.125782 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk"] Jul 17 18:56:14.126249 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:14.126233 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06de94b3-4c22-4dec-9265-e77e606da446" containerName="revoke-keys" Jul 17 18:56:14.126300 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:14.126251 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="06de94b3-4c22-4dec-9265-e77e606da446" containerName="revoke-keys" Jul 17 18:56:14.129152 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:14.129138 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk" Jul 17 18:56:14.139361 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:14.139332 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk"] Jul 17 18:56:14.182725 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:14.182680 2572 generic.go:356] "Generic (PLEG): container finished" podID="58c52746-88e6-4cd9-9732-701ef0386536" containerID="d47864388a1adcba65266cb0837c2931c47b038a1d6ec5c699cb7934c3c2a91e" exitCode=6 Jul 17 18:56:14.183159 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:14.182742 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc" event={"ID":"58c52746-88e6-4cd9-9732-701ef0386536","Type":"ContainerDied","Data":"d47864388a1adcba65266cb0837c2931c47b038a1d6ec5c699cb7934c3c2a91e"} Jul 17 18:56:14.183159 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:14.182806 2572 scope.go:117] "RemoveContainer" containerID="5824bf1c7e53b790742832456e5a89cbe911986d943e62fdd407c1954079d271" Jul 17 18:56:14.183159 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:14.182992 2572 scope.go:117] "RemoveContainer" containerID="d47864388a1adcba65266cb0837c2931c47b038a1d6ec5c699cb7934c3c2a91e" Jul 17 18:56:14.183313 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:56:14.183238 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc_odh-ai-gateway-infra(58c52746-88e6-4cd9-9732-701ef0386536)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc" podUID="58c52746-88e6-4cd9-9732-701ef0386536" Jul 17 18:56:14.184761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:14.184735 2572 generic.go:356] "Generic (PLEG): container finished" podID="1b05351a-2be2-4b43-a273-b6eddab39909" containerID="5f6dd853811d3306649d44424586c9d4667b3592db89ca3b397f538de389ff79" exitCode=6 Jul 17 18:56:14.184897 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:14.184770 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh" event={"ID":"1b05351a-2be2-4b43-a273-b6eddab39909","Type":"ContainerDied","Data":"5f6dd853811d3306649d44424586c9d4667b3592db89ca3b397f538de389ff79"} Jul 17 18:56:14.185071 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:14.185055 2572 scope.go:117] "RemoveContainer" containerID="5f6dd853811d3306649d44424586c9d4667b3592db89ca3b397f538de389ff79" Jul 17 18:56:14.185230 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:56:14.185210 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh_odh-ai-gateway-infra(1b05351a-2be2-4b43-a273-b6eddab39909)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh" podUID="1b05351a-2be2-4b43-a273-b6eddab39909" Jul 17 18:56:14.193916 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:14.193898 2572 scope.go:117] "RemoveContainer" containerID="09f09c09952da27ec279467902f5bbd09cc9de2c8d9518931c97f234b5b6ec16" Jul 17 18:56:14.227415 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:14.227394 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911c5b17-6b69-4207-acbe-a7965083ad17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk\" (UID: \"911c5b17-6b69-4207-acbe-a7965083ad17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk" Jul 17 18:56:14.328810 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:14.328778 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911c5b17-6b69-4207-acbe-a7965083ad17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk\" (UID: \"911c5b17-6b69-4207-acbe-a7965083ad17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk" Jul 17 18:56:14.329359 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:14.329343 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911c5b17-6b69-4207-acbe-a7965083ad17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk\" (UID: \"911c5b17-6b69-4207-acbe-a7965083ad17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk" Jul 17 18:56:14.439124 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:14.439039 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk" Jul 17 18:56:14.566002 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:14.565978 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk"] Jul 17 18:56:14.567620 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:56:14.567598 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911c5b17_6b69_4207_acbe_a7965083ad17.slice/crio-1249bb58c541f41ef334eb7a5dc2f4853b206ff0f6ff0c2acff80d139c1b37c0 WatchSource:0}: Error finding container 1249bb58c541f41ef334eb7a5dc2f4853b206ff0f6ff0c2acff80d139c1b37c0: Status 404 returned error can't find the container with id 1249bb58c541f41ef334eb7a5dc2f4853b206ff0f6ff0c2acff80d139c1b37c0 Jul 17 18:56:15.191196 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:15.191170 2572 generic.go:356] "Generic (PLEG): container finished" podID="911c5b17-6b69-4207-acbe-a7965083ad17" containerID="6b418324e002abbebabc1584cc6edb2d537a4a2a5814b329057473e2d7132bd9" exitCode=6 Jul 17 18:56:15.191608 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:15.191257 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk" event={"ID":"911c5b17-6b69-4207-acbe-a7965083ad17","Type":"ContainerDied","Data":"6b418324e002abbebabc1584cc6edb2d537a4a2a5814b329057473e2d7132bd9"} Jul 17 18:56:15.191608 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:15.191312 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk" event={"ID":"911c5b17-6b69-4207-acbe-a7965083ad17","Type":"ContainerStarted","Data":"1249bb58c541f41ef334eb7a5dc2f4853b206ff0f6ff0c2acff80d139c1b37c0"} Jul 17 18:56:15.191608 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:15.191418 2572 scope.go:117] "RemoveContainer" containerID="6b418324e002abbebabc1584cc6edb2d537a4a2a5814b329057473e2d7132bd9" Jul 17 18:56:15.193062 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:15.193045 2572 scope.go:117] "RemoveContainer" containerID="d47864388a1adcba65266cb0837c2931c47b038a1d6ec5c699cb7934c3c2a91e" Jul 17 18:56:15.193274 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:56:15.193241 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc_odh-ai-gateway-infra(58c52746-88e6-4cd9-9732-701ef0386536)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc" podUID="58c52746-88e6-4cd9-9732-701ef0386536" Jul 17 18:56:15.226364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:15.226342 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh"] Jul 17 18:56:15.367536 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:15.367514 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh" Jul 17 18:56:15.441867 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:15.441845 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b05351a-2be2-4b43-a273-b6eddab39909-maas-api-service-ca\") pod \"1b05351a-2be2-4b43-a273-b6eddab39909\" (UID: \"1b05351a-2be2-4b43-a273-b6eddab39909\") " Jul 17 18:56:15.442190 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:15.442167 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b05351a-2be2-4b43-a273-b6eddab39909-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b05351a-2be2-4b43-a273-b6eddab39909" (UID: "1b05351a-2be2-4b43-a273-b6eddab39909"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:56:15.542978 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:15.542942 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b05351a-2be2-4b43-a273-b6eddab39909-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:56:16.201140 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:16.201057 2572 generic.go:356] "Generic (PLEG): container finished" podID="911c5b17-6b69-4207-acbe-a7965083ad17" containerID="f42935c75f2a1a2d9c2ae0070340c1ac600e179c9f8b120a9a6329f85a9c5709" exitCode=6 Jul 17 18:56:16.201616 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:16.201146 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk" event={"ID":"911c5b17-6b69-4207-acbe-a7965083ad17","Type":"ContainerDied","Data":"f42935c75f2a1a2d9c2ae0070340c1ac600e179c9f8b120a9a6329f85a9c5709"} Jul 17 18:56:16.201616 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:16.201245 2572 scope.go:117] "RemoveContainer" containerID="6b418324e002abbebabc1584cc6edb2d537a4a2a5814b329057473e2d7132bd9" Jul 17 18:56:16.201616 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:16.201448 2572 scope.go:117] "RemoveContainer" containerID="f42935c75f2a1a2d9c2ae0070340c1ac600e179c9f8b120a9a6329f85a9c5709" Jul 17 18:56:16.201791 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:56:16.201693 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk_odh-ai-gateway-infra(911c5b17-6b69-4207-acbe-a7965083ad17)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk" podUID="911c5b17-6b69-4207-acbe-a7965083ad17" Jul 17 18:56:16.202479 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:16.202459 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh" event={"ID":"1b05351a-2be2-4b43-a273-b6eddab39909","Type":"ContainerDied","Data":"d31fead1612eadd1f650926f4c39d3343826849a1bc22420a896b53e0bbf4904"} Jul 17 18:56:16.202532 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:16.202466 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh" Jul 17 18:56:16.211882 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:16.211864 2572 scope.go:117] "RemoveContainer" containerID="5f6dd853811d3306649d44424586c9d4667b3592db89ca3b397f538de389ff79" Jul 17 18:56:16.229281 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:16.229256 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh"] Jul 17 18:56:16.232902 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:16.232883 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-6fsmh"] Jul 17 18:56:17.208328 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:17.208300 2572 scope.go:117] "RemoveContainer" containerID="f42935c75f2a1a2d9c2ae0070340c1ac600e179c9f8b120a9a6329f85a9c5709" Jul 17 18:56:17.208737 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:56:17.208537 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk_odh-ai-gateway-infra(911c5b17-6b69-4207-acbe-a7965083ad17)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk" podUID="911c5b17-6b69-4207-acbe-a7965083ad17" Jul 17 18:56:17.223897 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:17.223864 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b05351a-2be2-4b43-a273-b6eddab39909" path="/var/lib/kubelet/pods/1b05351a-2be2-4b43-a273-b6eddab39909/volumes" Jul 17 18:56:28.220210 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:28.220178 2572 scope.go:117] "RemoveContainer" containerID="d47864388a1adcba65266cb0837c2931c47b038a1d6ec5c699cb7934c3c2a91e" Jul 17 18:56:29.220411 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:29.220336 2572 scope.go:117] "RemoveContainer" containerID="f42935c75f2a1a2d9c2ae0070340c1ac600e179c9f8b120a9a6329f85a9c5709" Jul 17 18:56:29.262021 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:29.261982 2572 generic.go:356] "Generic (PLEG): container finished" podID="58c52746-88e6-4cd9-9732-701ef0386536" containerID="d7af4383330e6a0d5bf69534ba68658f01d1420d474993259742faa84e6dc011" exitCode=6 Jul 17 18:56:29.262135 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:29.262038 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc" event={"ID":"58c52746-88e6-4cd9-9732-701ef0386536","Type":"ContainerDied","Data":"d7af4383330e6a0d5bf69534ba68658f01d1420d474993259742faa84e6dc011"} Jul 17 18:56:29.262135 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:29.262098 2572 scope.go:117] "RemoveContainer" containerID="d47864388a1adcba65266cb0837c2931c47b038a1d6ec5c699cb7934c3c2a91e" Jul 17 18:56:29.262354 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:29.262341 2572 scope.go:117] "RemoveContainer" containerID="d7af4383330e6a0d5bf69534ba68658f01d1420d474993259742faa84e6dc011" Jul 17 18:56:29.262539 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:56:29.262522 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc_odh-ai-gateway-infra(58c52746-88e6-4cd9-9732-701ef0386536)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc" podUID="58c52746-88e6-4cd9-9732-701ef0386536" Jul 17 18:56:30.268059 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:30.267959 2572 generic.go:356] "Generic (PLEG): container finished" podID="911c5b17-6b69-4207-acbe-a7965083ad17" containerID="506af28f86d2d9ae971c8101a16fa72e561ba9a701c132f571161b35c4d8aae0" exitCode=6 Jul 17 18:56:30.268059 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:30.268035 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk" event={"ID":"911c5b17-6b69-4207-acbe-a7965083ad17","Type":"ContainerDied","Data":"506af28f86d2d9ae971c8101a16fa72e561ba9a701c132f571161b35c4d8aae0"} Jul 17 18:56:30.268549 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:30.268126 2572 scope.go:117] "RemoveContainer" containerID="f42935c75f2a1a2d9c2ae0070340c1ac600e179c9f8b120a9a6329f85a9c5709" Jul 17 18:56:30.268549 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:30.268475 2572 scope.go:117] "RemoveContainer" containerID="506af28f86d2d9ae971c8101a16fa72e561ba9a701c132f571161b35c4d8aae0" Jul 17 18:56:30.268739 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:56:30.268714 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk_odh-ai-gateway-infra(911c5b17-6b69-4207-acbe-a7965083ad17)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk" podUID="911c5b17-6b69-4207-acbe-a7965083ad17" Jul 17 18:56:30.288988 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:30.288961 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc"] Jul 17 18:56:30.420136 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:30.420114 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc" Jul 17 18:56:30.451602 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:30.451574 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c52746-88e6-4cd9-9732-701ef0386536-maas-api-service-ca\") pod \"58c52746-88e6-4cd9-9732-701ef0386536\" (UID: \"58c52746-88e6-4cd9-9732-701ef0386536\") " Jul 17 18:56:30.451886 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:30.451865 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c52746-88e6-4cd9-9732-701ef0386536-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "58c52746-88e6-4cd9-9732-701ef0386536" (UID: "58c52746-88e6-4cd9-9732-701ef0386536"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:56:30.552625 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:30.552596 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58c52746-88e6-4cd9-9732-701ef0386536-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:56:31.275257 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:31.275226 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc" Jul 17 18:56:31.275667 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:31.275257 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc" event={"ID":"58c52746-88e6-4cd9-9732-701ef0386536","Type":"ContainerDied","Data":"f3dc9dfb436f1eb5b85e6c690307d0e1b1f6e26a2a89447f9fcce38c4ee30bd3"} Jul 17 18:56:31.275667 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:31.275318 2572 scope.go:117] "RemoveContainer" containerID="d7af4383330e6a0d5bf69534ba68658f01d1420d474993259742faa84e6dc011" Jul 17 18:56:31.292415 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:31.292391 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc"] Jul 17 18:56:31.295508 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:31.295488 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk"] Jul 17 18:56:31.302643 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:31.302618 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-2njjc"] Jul 17 18:56:31.437380 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:31.437351 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk" Jul 17 18:56:31.459268 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:31.459247 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911c5b17-6b69-4207-acbe-a7965083ad17-maas-api-service-ca\") pod \"911c5b17-6b69-4207-acbe-a7965083ad17\" (UID: \"911c5b17-6b69-4207-acbe-a7965083ad17\") " Jul 17 18:56:31.459558 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:31.459537 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911c5b17-6b69-4207-acbe-a7965083ad17-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "911c5b17-6b69-4207-acbe-a7965083ad17" (UID: "911c5b17-6b69-4207-acbe-a7965083ad17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:56:31.560305 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:31.560277 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/911c5b17-6b69-4207-acbe-a7965083ad17-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:56:32.283708 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:32.283669 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk" event={"ID":"911c5b17-6b69-4207-acbe-a7965083ad17","Type":"ContainerDied","Data":"1249bb58c541f41ef334eb7a5dc2f4853b206ff0f6ff0c2acff80d139c1b37c0"} Jul 17 18:56:32.283708 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:32.283705 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk" Jul 17 18:56:32.284262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:32.283742 2572 scope.go:117] "RemoveContainer" containerID="506af28f86d2d9ae971c8101a16fa72e561ba9a701c132f571161b35c4d8aae0" Jul 17 18:56:32.306898 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:32.306873 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk"] Jul 17 18:56:32.311610 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:32.311588 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-bk6bk"] Jul 17 18:56:33.224839 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:33.224805 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58c52746-88e6-4cd9-9732-701ef0386536" path="/var/lib/kubelet/pods/58c52746-88e6-4cd9-9732-701ef0386536/volumes" Jul 17 18:56:33.225274 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:33.225256 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="911c5b17-6b69-4207-acbe-a7965083ad17" path="/var/lib/kubelet/pods/911c5b17-6b69-4207-acbe-a7965083ad17/volumes" Jul 17 18:56:38.125170 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125087 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z"] Jul 17 18:56:38.125544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125369 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="911c5b17-6b69-4207-acbe-a7965083ad17" containerName="revoke-keys" Jul 17 18:56:38.125544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125380 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="911c5b17-6b69-4207-acbe-a7965083ad17" containerName="revoke-keys" Jul 17 18:56:38.125544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125389 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b05351a-2be2-4b43-a273-b6eddab39909" containerName="revoke-keys" Jul 17 18:56:38.125544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125395 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b05351a-2be2-4b43-a273-b6eddab39909" containerName="revoke-keys" Jul 17 18:56:38.125544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125401 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="911c5b17-6b69-4207-acbe-a7965083ad17" containerName="revoke-keys" Jul 17 18:56:38.125544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125407 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="911c5b17-6b69-4207-acbe-a7965083ad17" containerName="revoke-keys" Jul 17 18:56:38.125544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125415 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b05351a-2be2-4b43-a273-b6eddab39909" containerName="revoke-keys" Jul 17 18:56:38.125544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125420 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b05351a-2be2-4b43-a273-b6eddab39909" containerName="revoke-keys" Jul 17 18:56:38.125544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125435 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c52746-88e6-4cd9-9732-701ef0386536" containerName="revoke-keys" Jul 17 18:56:38.125544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125440 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c52746-88e6-4cd9-9732-701ef0386536" containerName="revoke-keys" Jul 17 18:56:38.125544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125448 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c52746-88e6-4cd9-9732-701ef0386536" containerName="revoke-keys" Jul 17 18:56:38.125544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125453 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c52746-88e6-4cd9-9732-701ef0386536" containerName="revoke-keys" Jul 17 18:56:38.125544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125458 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c52746-88e6-4cd9-9732-701ef0386536" containerName="revoke-keys" Jul 17 18:56:38.125544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125463 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c52746-88e6-4cd9-9732-701ef0386536" containerName="revoke-keys" Jul 17 18:56:38.125544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125519 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b05351a-2be2-4b43-a273-b6eddab39909" containerName="revoke-keys" Jul 17 18:56:38.125544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125544 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b05351a-2be2-4b43-a273-b6eddab39909" containerName="revoke-keys" Jul 17 18:56:38.125544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125551 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="911c5b17-6b69-4207-acbe-a7965083ad17" containerName="revoke-keys" Jul 17 18:56:38.126134 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125557 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="58c52746-88e6-4cd9-9732-701ef0386536" containerName="revoke-keys" Jul 17 18:56:38.126134 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125563 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="911c5b17-6b69-4207-acbe-a7965083ad17" containerName="revoke-keys" Jul 17 18:56:38.126134 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125570 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="58c52746-88e6-4cd9-9732-701ef0386536" containerName="revoke-keys" Jul 17 18:56:38.126134 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.125575 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="58c52746-88e6-4cd9-9732-701ef0386536" containerName="revoke-keys" Jul 17 18:56:38.129729 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.129713 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z" Jul 17 18:56:38.132265 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.132236 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:56:38.132389 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.132340 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:56:38.135851 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.135823 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z"] Jul 17 18:56:38.216801 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.216765 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2539ae74-d22a-4206-ba51-63552374aa79-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z\" (UID: \"2539ae74-d22a-4206-ba51-63552374aa79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z" Jul 17 18:56:38.317780 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.317748 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2539ae74-d22a-4206-ba51-63552374aa79-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z\" (UID: \"2539ae74-d22a-4206-ba51-63552374aa79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z" Jul 17 18:56:38.318388 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.318372 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2539ae74-d22a-4206-ba51-63552374aa79-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z\" (UID: \"2539ae74-d22a-4206-ba51-63552374aa79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z" Jul 17 18:56:38.441115 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.441039 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z" Jul 17 18:56:38.564749 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:38.564725 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z"] Jul 17 18:56:38.566601 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:56:38.566575 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2539ae74_d22a_4206_ba51_63552374aa79.slice/crio-2046a4eeff991989a1a64625fa908a1cf00ba7db94d8511398314254cde41463 WatchSource:0}: Error finding container 2046a4eeff991989a1a64625fa908a1cf00ba7db94d8511398314254cde41463: Status 404 returned error can't find the container with id 2046a4eeff991989a1a64625fa908a1cf00ba7db94d8511398314254cde41463 Jul 17 18:56:39.318051 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:39.317998 2572 generic.go:356] "Generic (PLEG): container finished" podID="2539ae74-d22a-4206-ba51-63552374aa79" containerID="c8e855269e395f6086ed23e5fde54cb7cdfe3a0f7e764b4907a0c284438f5fe1" exitCode=6 Jul 17 18:56:39.318446 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:39.318085 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z" event={"ID":"2539ae74-d22a-4206-ba51-63552374aa79","Type":"ContainerDied","Data":"c8e855269e395f6086ed23e5fde54cb7cdfe3a0f7e764b4907a0c284438f5fe1"} Jul 17 18:56:39.318446 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:39.318146 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z" event={"ID":"2539ae74-d22a-4206-ba51-63552374aa79","Type":"ContainerStarted","Data":"2046a4eeff991989a1a64625fa908a1cf00ba7db94d8511398314254cde41463"} Jul 17 18:56:39.318446 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:39.318247 2572 scope.go:117] "RemoveContainer" containerID="c8e855269e395f6086ed23e5fde54cb7cdfe3a0f7e764b4907a0c284438f5fe1" Jul 17 18:56:40.324456 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:40.324425 2572 generic.go:356] "Generic (PLEG): container finished" podID="2539ae74-d22a-4206-ba51-63552374aa79" containerID="1d06e9160d2a331f00f2d0e7794ae1f5e75d361e8df4729da98ed9fe66bb0e5a" exitCode=6 Jul 17 18:56:40.324858 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:40.324513 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z" event={"ID":"2539ae74-d22a-4206-ba51-63552374aa79","Type":"ContainerDied","Data":"1d06e9160d2a331f00f2d0e7794ae1f5e75d361e8df4729da98ed9fe66bb0e5a"} Jul 17 18:56:40.324858 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:40.324576 2572 scope.go:117] "RemoveContainer" containerID="c8e855269e395f6086ed23e5fde54cb7cdfe3a0f7e764b4907a0c284438f5fe1" Jul 17 18:56:40.324858 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:40.324795 2572 scope.go:117] "RemoveContainer" containerID="1d06e9160d2a331f00f2d0e7794ae1f5e75d361e8df4729da98ed9fe66bb0e5a" Jul 17 18:56:40.325049 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:56:40.325031 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z_odh-ai-gateway-infra(2539ae74-d22a-4206-ba51-63552374aa79)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z" podUID="2539ae74-d22a-4206-ba51-63552374aa79" Jul 17 18:56:41.331172 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:41.331144 2572 scope.go:117] "RemoveContainer" containerID="1d06e9160d2a331f00f2d0e7794ae1f5e75d361e8df4729da98ed9fe66bb0e5a" Jul 17 18:56:41.331534 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:56:41.331323 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z_odh-ai-gateway-infra(2539ae74-d22a-4206-ba51-63552374aa79)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z" podUID="2539ae74-d22a-4206-ba51-63552374aa79" Jul 17 18:56:53.219529 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:53.219490 2572 scope.go:117] "RemoveContainer" containerID="1d06e9160d2a331f00f2d0e7794ae1f5e75d361e8df4729da98ed9fe66bb0e5a" Jul 17 18:56:54.388727 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:54.388694 2572 generic.go:356] "Generic (PLEG): container finished" podID="2539ae74-d22a-4206-ba51-63552374aa79" containerID="9a55bb87069b6251cac7aed4b659f62eb1ea90fa29b4de6ce01b8beda3388ca2" exitCode=6 Jul 17 18:56:54.389159 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:54.388762 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z" event={"ID":"2539ae74-d22a-4206-ba51-63552374aa79","Type":"ContainerDied","Data":"9a55bb87069b6251cac7aed4b659f62eb1ea90fa29b4de6ce01b8beda3388ca2"} Jul 17 18:56:54.389159 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:54.388838 2572 scope.go:117] "RemoveContainer" containerID="1d06e9160d2a331f00f2d0e7794ae1f5e75d361e8df4729da98ed9fe66bb0e5a" Jul 17 18:56:54.389261 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:54.389190 2572 scope.go:117] "RemoveContainer" containerID="9a55bb87069b6251cac7aed4b659f62eb1ea90fa29b4de6ce01b8beda3388ca2" Jul 17 18:56:54.389449 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:56:54.389430 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z_odh-ai-gateway-infra(2539ae74-d22a-4206-ba51-63552374aa79)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z" podUID="2539ae74-d22a-4206-ba51-63552374aa79" Jul 17 18:56:55.415351 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:55.415321 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z"] Jul 17 18:56:55.543050 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:55.543030 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z" Jul 17 18:56:55.680005 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:55.679926 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2539ae74-d22a-4206-ba51-63552374aa79-maas-api-service-ca\") pod \"2539ae74-d22a-4206-ba51-63552374aa79\" (UID: \"2539ae74-d22a-4206-ba51-63552374aa79\") " Jul 17 18:56:55.680302 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:55.680280 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2539ae74-d22a-4206-ba51-63552374aa79-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2539ae74-d22a-4206-ba51-63552374aa79" (UID: "2539ae74-d22a-4206-ba51-63552374aa79"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:56:55.781363 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:55.781336 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2539ae74-d22a-4206-ba51-63552374aa79-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:56:56.005106 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.005028 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c"] Jul 17 18:56:56.005323 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.005311 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2539ae74-d22a-4206-ba51-63552374aa79" containerName="revoke-keys" Jul 17 18:56:56.005365 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.005324 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2539ae74-d22a-4206-ba51-63552374aa79" containerName="revoke-keys" Jul 17 18:56:56.005365 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.005337 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2539ae74-d22a-4206-ba51-63552374aa79" containerName="revoke-keys" Jul 17 18:56:56.005365 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.005342 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2539ae74-d22a-4206-ba51-63552374aa79" containerName="revoke-keys" Jul 17 18:56:56.005365 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.005358 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="911c5b17-6b69-4207-acbe-a7965083ad17" containerName="revoke-keys" Jul 17 18:56:56.005365 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.005363 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="911c5b17-6b69-4207-acbe-a7965083ad17" containerName="revoke-keys" Jul 17 18:56:56.005535 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.005369 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b05351a-2be2-4b43-a273-b6eddab39909" containerName="revoke-keys" Jul 17 18:56:56.005535 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.005374 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b05351a-2be2-4b43-a273-b6eddab39909" containerName="revoke-keys" Jul 17 18:56:56.005535 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.005416 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2539ae74-d22a-4206-ba51-63552374aa79" containerName="revoke-keys" Jul 17 18:56:56.005535 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.005422 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2539ae74-d22a-4206-ba51-63552374aa79" containerName="revoke-keys" Jul 17 18:56:56.005535 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.005428 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2539ae74-d22a-4206-ba51-63552374aa79" containerName="revoke-keys" Jul 17 18:56:56.005535 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.005435 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b05351a-2be2-4b43-a273-b6eddab39909" containerName="revoke-keys" Jul 17 18:56:56.005535 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.005441 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="911c5b17-6b69-4207-acbe-a7965083ad17" containerName="revoke-keys" Jul 17 18:56:56.008386 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.008370 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c" Jul 17 18:56:56.015763 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.015727 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c"] Jul 17 18:56:56.084938 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.084903 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81ecb2fa-1902-4715-b873-11030369dca2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c\" (UID: \"81ecb2fa-1902-4715-b873-11030369dca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c" Jul 17 18:56:56.186261 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.186232 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81ecb2fa-1902-4715-b873-11030369dca2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c\" (UID: \"81ecb2fa-1902-4715-b873-11030369dca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c" Jul 17 18:56:56.186793 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.186775 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81ecb2fa-1902-4715-b873-11030369dca2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c\" (UID: \"81ecb2fa-1902-4715-b873-11030369dca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c" Jul 17 18:56:56.319979 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.319953 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c" Jul 17 18:56:56.400733 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.400662 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z" event={"ID":"2539ae74-d22a-4206-ba51-63552374aa79","Type":"ContainerDied","Data":"2046a4eeff991989a1a64625fa908a1cf00ba7db94d8511398314254cde41463"} Jul 17 18:56:56.400733 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.400699 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z" Jul 17 18:56:56.400733 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.400730 2572 scope.go:117] "RemoveContainer" containerID="9a55bb87069b6251cac7aed4b659f62eb1ea90fa29b4de6ce01b8beda3388ca2" Jul 17 18:56:56.430282 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.430257 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z"] Jul 17 18:56:56.439385 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.439358 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-ktn2z"] Jul 17 18:56:56.451634 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:56.451614 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c"] Jul 17 18:56:56.453597 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:56:56.453573 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ecb2fa_1902_4715_b873_11030369dca2.slice/crio-2c0a70bd506bf204ce1cc8a8cdfc6d2324b550cf79596c9ecac6fcb9211b2369 WatchSource:0}: Error finding container 2c0a70bd506bf204ce1cc8a8cdfc6d2324b550cf79596c9ecac6fcb9211b2369: Status 404 returned error can't find the container with id 2c0a70bd506bf204ce1cc8a8cdfc6d2324b550cf79596c9ecac6fcb9211b2369 Jul 17 18:56:57.224384 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:57.224354 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2539ae74-d22a-4206-ba51-63552374aa79" path="/var/lib/kubelet/pods/2539ae74-d22a-4206-ba51-63552374aa79/volumes" Jul 17 18:56:57.407058 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:57.406957 2572 generic.go:356] "Generic (PLEG): container finished" podID="81ecb2fa-1902-4715-b873-11030369dca2" containerID="bcf7eef58f36de5f8e0335bea67413ce697f371af0757f5389113999696d8b52" exitCode=6 Jul 17 18:56:57.407058 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:57.406989 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c" event={"ID":"81ecb2fa-1902-4715-b873-11030369dca2","Type":"ContainerDied","Data":"bcf7eef58f36de5f8e0335bea67413ce697f371af0757f5389113999696d8b52"} Jul 17 18:56:57.407058 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:57.407050 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c" event={"ID":"81ecb2fa-1902-4715-b873-11030369dca2","Type":"ContainerStarted","Data":"2c0a70bd506bf204ce1cc8a8cdfc6d2324b550cf79596c9ecac6fcb9211b2369"} Jul 17 18:56:57.407332 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:57.407211 2572 scope.go:117] "RemoveContainer" containerID="bcf7eef58f36de5f8e0335bea67413ce697f371af0757f5389113999696d8b52" Jul 17 18:56:58.413144 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:58.413053 2572 generic.go:356] "Generic (PLEG): container finished" podID="81ecb2fa-1902-4715-b873-11030369dca2" containerID="dc116554b23f0214dfc4cfa8f3a0add47bcc3e7576c442654eb8ba1ec15c5bca" exitCode=6 Jul 17 18:56:58.413144 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:58.413102 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c" event={"ID":"81ecb2fa-1902-4715-b873-11030369dca2","Type":"ContainerDied","Data":"dc116554b23f0214dfc4cfa8f3a0add47bcc3e7576c442654eb8ba1ec15c5bca"} Jul 17 18:56:58.413654 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:58.413163 2572 scope.go:117] "RemoveContainer" containerID="bcf7eef58f36de5f8e0335bea67413ce697f371af0757f5389113999696d8b52" Jul 17 18:56:58.413654 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:58.413402 2572 scope.go:117] "RemoveContainer" containerID="dc116554b23f0214dfc4cfa8f3a0add47bcc3e7576c442654eb8ba1ec15c5bca" Jul 17 18:56:58.413654 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:56:58.413623 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c_odh-ai-gateway-infra(81ecb2fa-1902-4715-b873-11030369dca2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c" podUID="81ecb2fa-1902-4715-b873-11030369dca2" Jul 17 18:56:59.418744 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:56:59.418719 2572 scope.go:117] "RemoveContainer" containerID="dc116554b23f0214dfc4cfa8f3a0add47bcc3e7576c442654eb8ba1ec15c5bca" Jul 17 18:56:59.419242 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:56:59.418908 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c_odh-ai-gateway-infra(81ecb2fa-1902-4715-b873-11030369dca2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c" podUID="81ecb2fa-1902-4715-b873-11030369dca2" Jul 17 18:57:01.608712 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:01.608677 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv"] Jul 17 18:57:01.609107 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:01.609002 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2539ae74-d22a-4206-ba51-63552374aa79" containerName="revoke-keys" Jul 17 18:57:01.609107 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:01.609026 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2539ae74-d22a-4206-ba51-63552374aa79" containerName="revoke-keys" Jul 17 18:57:01.613278 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:01.613259 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv" Jul 17 18:57:01.619566 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:01.619541 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv"] Jul 17 18:57:01.736645 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:01.736605 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2644b523-29f2-497c-9bf8-5ca6e9294523-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv\" (UID: \"2644b523-29f2-497c-9bf8-5ca6e9294523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv" Jul 17 18:57:01.837957 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:01.837922 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2644b523-29f2-497c-9bf8-5ca6e9294523-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv\" (UID: \"2644b523-29f2-497c-9bf8-5ca6e9294523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv" Jul 17 18:57:01.838513 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:01.838494 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2644b523-29f2-497c-9bf8-5ca6e9294523-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv\" (UID: \"2644b523-29f2-497c-9bf8-5ca6e9294523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv" Jul 17 18:57:01.924377 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:01.924308 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv" Jul 17 18:57:02.046533 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:02.046513 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv"] Jul 17 18:57:02.048360 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:57:02.048327 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2644b523_29f2_497c_9bf8_5ca6e9294523.slice/crio-659eabc2f7efab020cf3a0d15b26303133271067bb24b831a9173c702a6551b0 WatchSource:0}: Error finding container 659eabc2f7efab020cf3a0d15b26303133271067bb24b831a9173c702a6551b0: Status 404 returned error can't find the container with id 659eabc2f7efab020cf3a0d15b26303133271067bb24b831a9173c702a6551b0 Jul 17 18:57:02.433504 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:02.433460 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv" event={"ID":"2644b523-29f2-497c-9bf8-5ca6e9294523","Type":"ContainerStarted","Data":"3827c9871262868787bd1ce4e0e7d30ea579fe6612dfa10f31a8384e663282b2"} Jul 17 18:57:02.433708 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:02.433525 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv" event={"ID":"2644b523-29f2-497c-9bf8-5ca6e9294523","Type":"ContainerStarted","Data":"659eabc2f7efab020cf3a0d15b26303133271067bb24b831a9173c702a6551b0"} Jul 17 18:57:02.448879 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:02.448830 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv" podStartSLOduration=1.448815353 podStartE2EDuration="1.448815353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:57:02.447279405 +0000 UTC m=+3617.856707011" watchObservedRunningTime="2026-07-17 18:57:02.448815353 +0000 UTC m=+3617.858242959" Jul 17 18:57:03.439212 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:03.439178 2572 generic.go:356] "Generic (PLEG): container finished" podID="2644b523-29f2-497c-9bf8-5ca6e9294523" containerID="3827c9871262868787bd1ce4e0e7d30ea579fe6612dfa10f31a8384e663282b2" exitCode=6 Jul 17 18:57:03.439678 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:03.439230 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv" event={"ID":"2644b523-29f2-497c-9bf8-5ca6e9294523","Type":"ContainerDied","Data":"3827c9871262868787bd1ce4e0e7d30ea579fe6612dfa10f31a8384e663282b2"} Jul 17 18:57:03.439678 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:03.439491 2572 scope.go:117] "RemoveContainer" containerID="3827c9871262868787bd1ce4e0e7d30ea579fe6612dfa10f31a8384e663282b2" Jul 17 18:57:04.169464 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:04.169437 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh"] Jul 17 18:57:04.172710 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:04.172694 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh" Jul 17 18:57:04.179393 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:04.179368 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh"] Jul 17 18:57:04.258538 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:04.258512 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4607669-f2d9-4ff1-8da0-9c0c19a3a570-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh\" (UID: \"d4607669-f2d9-4ff1-8da0-9c0c19a3a570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh" Jul 17 18:57:04.359299 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:04.359272 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4607669-f2d9-4ff1-8da0-9c0c19a3a570-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh\" (UID: \"d4607669-f2d9-4ff1-8da0-9c0c19a3a570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh" Jul 17 18:57:04.359923 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:04.359905 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4607669-f2d9-4ff1-8da0-9c0c19a3a570-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh\" (UID: \"d4607669-f2d9-4ff1-8da0-9c0c19a3a570\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh" Jul 17 18:57:04.445300 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:04.445214 2572 generic.go:356] "Generic (PLEG): container finished" podID="2644b523-29f2-497c-9bf8-5ca6e9294523" containerID="8e2ddce215323736d392c65e6ac6899a39a9f49648009159ff87bf813fa0b11e" exitCode=6 Jul 17 18:57:04.445665 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:04.445295 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv" event={"ID":"2644b523-29f2-497c-9bf8-5ca6e9294523","Type":"ContainerDied","Data":"8e2ddce215323736d392c65e6ac6899a39a9f49648009159ff87bf813fa0b11e"} Jul 17 18:57:04.445665 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:04.445364 2572 scope.go:117] "RemoveContainer" containerID="3827c9871262868787bd1ce4e0e7d30ea579fe6612dfa10f31a8384e663282b2" Jul 17 18:57:04.445665 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:04.445620 2572 scope.go:117] "RemoveContainer" containerID="8e2ddce215323736d392c65e6ac6899a39a9f49648009159ff87bf813fa0b11e" Jul 17 18:57:04.445855 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:57:04.445836 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv_odh-ai-gateway-infra(2644b523-29f2-497c-9bf8-5ca6e9294523)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv" podUID="2644b523-29f2-497c-9bf8-5ca6e9294523" Jul 17 18:57:04.508590 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:04.508561 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh" Jul 17 18:57:04.838287 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:04.838259 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh"] Jul 17 18:57:04.839840 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:57:04.839814 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4607669_f2d9_4ff1_8da0_9c0c19a3a570.slice/crio-877ce343a63f25955bc801f6af31fb7504dd9592953c64fa35511cb1e91eba3f WatchSource:0}: Error finding container 877ce343a63f25955bc801f6af31fb7504dd9592953c64fa35511cb1e91eba3f: Status 404 returned error can't find the container with id 877ce343a63f25955bc801f6af31fb7504dd9592953c64fa35511cb1e91eba3f Jul 17 18:57:05.451730 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:05.451701 2572 generic.go:356] "Generic (PLEG): container finished" podID="d4607669-f2d9-4ff1-8da0-9c0c19a3a570" containerID="0a983945d538336c9afc9892725c433b49be0d5aced216619fcabc003125c524" exitCode=6 Jul 17 18:57:05.452101 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:05.451781 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh" event={"ID":"d4607669-f2d9-4ff1-8da0-9c0c19a3a570","Type":"ContainerDied","Data":"0a983945d538336c9afc9892725c433b49be0d5aced216619fcabc003125c524"} Jul 17 18:57:05.452101 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:05.451836 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh" event={"ID":"d4607669-f2d9-4ff1-8da0-9c0c19a3a570","Type":"ContainerStarted","Data":"877ce343a63f25955bc801f6af31fb7504dd9592953c64fa35511cb1e91eba3f"} Jul 17 18:57:05.452101 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:05.451981 2572 scope.go:117] "RemoveContainer" containerID="0a983945d538336c9afc9892725c433b49be0d5aced216619fcabc003125c524" Jul 17 18:57:05.453723 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:05.453704 2572 scope.go:117] "RemoveContainer" containerID="8e2ddce215323736d392c65e6ac6899a39a9f49648009159ff87bf813fa0b11e" Jul 17 18:57:05.453938 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:57:05.453916 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv_odh-ai-gateway-infra(2644b523-29f2-497c-9bf8-5ca6e9294523)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv" podUID="2644b523-29f2-497c-9bf8-5ca6e9294523" Jul 17 18:57:06.459345 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:06.459263 2572 generic.go:356] "Generic (PLEG): container finished" podID="d4607669-f2d9-4ff1-8da0-9c0c19a3a570" containerID="e906d7bfeaa35064cbbfe4868136cd56308160f05670d72c1ab78db0f6922944" exitCode=6 Jul 17 18:57:06.459767 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:06.459349 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh" event={"ID":"d4607669-f2d9-4ff1-8da0-9c0c19a3a570","Type":"ContainerDied","Data":"e906d7bfeaa35064cbbfe4868136cd56308160f05670d72c1ab78db0f6922944"} Jul 17 18:57:06.459767 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:06.459414 2572 scope.go:117] "RemoveContainer" containerID="0a983945d538336c9afc9892725c433b49be0d5aced216619fcabc003125c524" Jul 17 18:57:06.459767 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:06.459580 2572 scope.go:117] "RemoveContainer" containerID="e906d7bfeaa35064cbbfe4868136cd56308160f05670d72c1ab78db0f6922944" Jul 17 18:57:06.459897 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:57:06.459820 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh_odh-ai-gateway-infra(d4607669-f2d9-4ff1-8da0-9c0c19a3a570)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh" podUID="d4607669-f2d9-4ff1-8da0-9c0c19a3a570" Jul 17 18:57:07.465162 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:07.465136 2572 scope.go:117] "RemoveContainer" containerID="e906d7bfeaa35064cbbfe4868136cd56308160f05670d72c1ab78db0f6922944" Jul 17 18:57:07.465563 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:57:07.465318 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh_odh-ai-gateway-infra(d4607669-f2d9-4ff1-8da0-9c0c19a3a570)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh" podUID="d4607669-f2d9-4ff1-8da0-9c0c19a3a570" Jul 17 18:57:10.220251 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:10.220220 2572 scope.go:117] "RemoveContainer" containerID="dc116554b23f0214dfc4cfa8f3a0add47bcc3e7576c442654eb8ba1ec15c5bca" Jul 17 18:57:11.483456 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:11.483424 2572 generic.go:356] "Generic (PLEG): container finished" podID="81ecb2fa-1902-4715-b873-11030369dca2" containerID="28c7b5a811381286d9009b8ae5c5d8988102929a9f2c411910011c97f95ca36b" exitCode=6 Jul 17 18:57:11.483456 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:11.483461 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c" event={"ID":"81ecb2fa-1902-4715-b873-11030369dca2","Type":"ContainerDied","Data":"28c7b5a811381286d9009b8ae5c5d8988102929a9f2c411910011c97f95ca36b"} Jul 17 18:57:11.483940 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:11.483516 2572 scope.go:117] "RemoveContainer" containerID="dc116554b23f0214dfc4cfa8f3a0add47bcc3e7576c442654eb8ba1ec15c5bca" Jul 17 18:57:11.483940 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:11.483824 2572 scope.go:117] "RemoveContainer" containerID="28c7b5a811381286d9009b8ae5c5d8988102929a9f2c411910011c97f95ca36b" Jul 17 18:57:11.484096 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:57:11.484077 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c_odh-ai-gateway-infra(81ecb2fa-1902-4715-b873-11030369dca2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c" podUID="81ecb2fa-1902-4715-b873-11030369dca2" Jul 17 18:57:12.516686 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:12.516654 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c"] Jul 17 18:57:12.648905 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:12.648883 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c" Jul 17 18:57:12.728589 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:12.728564 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81ecb2fa-1902-4715-b873-11030369dca2-maas-api-service-ca\") pod \"81ecb2fa-1902-4715-b873-11030369dca2\" (UID: \"81ecb2fa-1902-4715-b873-11030369dca2\") " Jul 17 18:57:12.728911 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:12.728889 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ecb2fa-1902-4715-b873-11030369dca2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "81ecb2fa-1902-4715-b873-11030369dca2" (UID: "81ecb2fa-1902-4715-b873-11030369dca2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:57:12.829907 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:12.829879 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81ecb2fa-1902-4715-b873-11030369dca2-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:57:13.494600 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:13.494523 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c" Jul 17 18:57:13.494600 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:13.494557 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c" event={"ID":"81ecb2fa-1902-4715-b873-11030369dca2","Type":"ContainerDied","Data":"2c0a70bd506bf204ce1cc8a8cdfc6d2324b550cf79596c9ecac6fcb9211b2369"} Jul 17 18:57:13.494797 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:13.494623 2572 scope.go:117] "RemoveContainer" containerID="28c7b5a811381286d9009b8ae5c5d8988102929a9f2c411910011c97f95ca36b" Jul 17 18:57:13.512179 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:13.512155 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c"] Jul 17 18:57:13.514443 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:13.514419 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-mj52c"] Jul 17 18:57:15.224892 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:15.224861 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81ecb2fa-1902-4715-b873-11030369dca2" path="/var/lib/kubelet/pods/81ecb2fa-1902-4715-b873-11030369dca2/volumes" Jul 17 18:57:20.220266 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:20.220238 2572 scope.go:117] "RemoveContainer" containerID="8e2ddce215323736d392c65e6ac6899a39a9f49648009159ff87bf813fa0b11e" Jul 17 18:57:21.531697 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:21.531660 2572 generic.go:356] "Generic (PLEG): container finished" podID="2644b523-29f2-497c-9bf8-5ca6e9294523" containerID="a240e314d657d6e3d07ca46cf05598282cb01652f8795f751d667cc793072c95" exitCode=6 Jul 17 18:57:21.532204 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:21.531733 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv" event={"ID":"2644b523-29f2-497c-9bf8-5ca6e9294523","Type":"ContainerDied","Data":"a240e314d657d6e3d07ca46cf05598282cb01652f8795f751d667cc793072c95"} Jul 17 18:57:21.532204 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:21.531792 2572 scope.go:117] "RemoveContainer" containerID="8e2ddce215323736d392c65e6ac6899a39a9f49648009159ff87bf813fa0b11e" Jul 17 18:57:21.532204 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:21.532159 2572 scope.go:117] "RemoveContainer" containerID="a240e314d657d6e3d07ca46cf05598282cb01652f8795f751d667cc793072c95" Jul 17 18:57:21.532436 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:57:21.532413 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv_odh-ai-gateway-infra(2644b523-29f2-497c-9bf8-5ca6e9294523)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv" podUID="2644b523-29f2-497c-9bf8-5ca6e9294523" Jul 17 18:57:22.220047 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:22.219998 2572 scope.go:117] "RemoveContainer" containerID="e906d7bfeaa35064cbbfe4868136cd56308160f05670d72c1ab78db0f6922944" Jul 17 18:57:22.566087 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:22.566058 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv"] Jul 17 18:57:22.703613 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:22.703589 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv" Jul 17 18:57:22.816167 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:22.816088 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2644b523-29f2-497c-9bf8-5ca6e9294523-maas-api-service-ca\") pod \"2644b523-29f2-497c-9bf8-5ca6e9294523\" (UID: \"2644b523-29f2-497c-9bf8-5ca6e9294523\") " Jul 17 18:57:22.816425 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:22.816404 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2644b523-29f2-497c-9bf8-5ca6e9294523-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2644b523-29f2-497c-9bf8-5ca6e9294523" (UID: "2644b523-29f2-497c-9bf8-5ca6e9294523"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:57:22.916670 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:22.916638 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2644b523-29f2-497c-9bf8-5ca6e9294523-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:57:23.543854 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:23.543828 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv" Jul 17 18:57:23.544052 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:23.543867 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv" event={"ID":"2644b523-29f2-497c-9bf8-5ca6e9294523","Type":"ContainerDied","Data":"659eabc2f7efab020cf3a0d15b26303133271067bb24b831a9173c702a6551b0"} Jul 17 18:57:23.544052 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:23.543968 2572 scope.go:117] "RemoveContainer" containerID="a240e314d657d6e3d07ca46cf05598282cb01652f8795f751d667cc793072c95" Jul 17 18:57:23.545971 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:23.545943 2572 generic.go:356] "Generic (PLEG): container finished" podID="d4607669-f2d9-4ff1-8da0-9c0c19a3a570" containerID="4a08f1999eb6d5a58182031e8843b96b18162d3ec43cf94d12102bd1cb5e4365" exitCode=6 Jul 17 18:57:23.546138 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:23.546053 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh" event={"ID":"d4607669-f2d9-4ff1-8da0-9c0c19a3a570","Type":"ContainerDied","Data":"4a08f1999eb6d5a58182031e8843b96b18162d3ec43cf94d12102bd1cb5e4365"} Jul 17 18:57:23.546396 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:23.546370 2572 scope.go:117] "RemoveContainer" containerID="4a08f1999eb6d5a58182031e8843b96b18162d3ec43cf94d12102bd1cb5e4365" Jul 17 18:57:23.546590 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:57:23.546571 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh_odh-ai-gateway-infra(d4607669-f2d9-4ff1-8da0-9c0c19a3a570)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh" podUID="d4607669-f2d9-4ff1-8da0-9c0c19a3a570" Jul 17 18:57:23.552230 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:23.552214 2572 scope.go:117] "RemoveContainer" containerID="e906d7bfeaa35064cbbfe4868136cd56308160f05670d72c1ab78db0f6922944" Jul 17 18:57:23.561892 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:23.561870 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv"] Jul 17 18:57:23.567505 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:23.567482 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-rchbv"] Jul 17 18:57:24.600425 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:24.600392 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh"] Jul 17 18:57:24.725646 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:24.725624 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh" Jul 17 18:57:24.832293 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:24.832265 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4607669-f2d9-4ff1-8da0-9c0c19a3a570-maas-api-service-ca\") pod \"d4607669-f2d9-4ff1-8da0-9c0c19a3a570\" (UID: \"d4607669-f2d9-4ff1-8da0-9c0c19a3a570\") " Jul 17 18:57:24.832646 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:24.832614 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4607669-f2d9-4ff1-8da0-9c0c19a3a570-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4607669-f2d9-4ff1-8da0-9c0c19a3a570" (UID: "d4607669-f2d9-4ff1-8da0-9c0c19a3a570"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:57:24.933518 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:24.933435 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4607669-f2d9-4ff1-8da0-9c0c19a3a570-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:57:25.224671 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:25.224591 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2644b523-29f2-497c-9bf8-5ca6e9294523" path="/var/lib/kubelet/pods/2644b523-29f2-497c-9bf8-5ca6e9294523/volumes" Jul 17 18:57:25.557234 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:25.557208 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh" Jul 17 18:57:25.557234 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:25.557218 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh" event={"ID":"d4607669-f2d9-4ff1-8da0-9c0c19a3a570","Type":"ContainerDied","Data":"877ce343a63f25955bc801f6af31fb7504dd9592953c64fa35511cb1e91eba3f"} Jul 17 18:57:25.557489 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:25.557284 2572 scope.go:117] "RemoveContainer" containerID="4a08f1999eb6d5a58182031e8843b96b18162d3ec43cf94d12102bd1cb5e4365" Jul 17 18:57:25.581280 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:25.581254 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh"] Jul 17 18:57:25.586806 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:25.586784 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-fc6rh"] Jul 17 18:57:27.223853 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:27.223821 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4607669-f2d9-4ff1-8da0-9c0c19a3a570" path="/var/lib/kubelet/pods/d4607669-f2d9-4ff1-8da0-9c0c19a3a570/volumes" Jul 17 18:57:28.170366 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170332 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m"] Jul 17 18:57:28.170626 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170614 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2644b523-29f2-497c-9bf8-5ca6e9294523" containerName="revoke-keys" Jul 17 18:57:28.170670 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170627 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2644b523-29f2-497c-9bf8-5ca6e9294523" containerName="revoke-keys" Jul 17 18:57:28.170670 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170638 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2644b523-29f2-497c-9bf8-5ca6e9294523" containerName="revoke-keys" Jul 17 18:57:28.170670 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170644 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2644b523-29f2-497c-9bf8-5ca6e9294523" containerName="revoke-keys" Jul 17 18:57:28.170670 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170651 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81ecb2fa-1902-4715-b873-11030369dca2" containerName="revoke-keys" Jul 17 18:57:28.170670 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170656 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="81ecb2fa-1902-4715-b873-11030369dca2" containerName="revoke-keys" Jul 17 18:57:28.170670 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170666 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81ecb2fa-1902-4715-b873-11030369dca2" containerName="revoke-keys" Jul 17 18:57:28.170859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170673 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="81ecb2fa-1902-4715-b873-11030369dca2" containerName="revoke-keys" Jul 17 18:57:28.170859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170682 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81ecb2fa-1902-4715-b873-11030369dca2" containerName="revoke-keys" Jul 17 18:57:28.170859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170687 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="81ecb2fa-1902-4715-b873-11030369dca2" containerName="revoke-keys" Jul 17 18:57:28.170859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170696 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4607669-f2d9-4ff1-8da0-9c0c19a3a570" containerName="revoke-keys" Jul 17 18:57:28.170859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170701 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4607669-f2d9-4ff1-8da0-9c0c19a3a570" containerName="revoke-keys" Jul 17 18:57:28.170859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170709 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4607669-f2d9-4ff1-8da0-9c0c19a3a570" containerName="revoke-keys" Jul 17 18:57:28.170859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170713 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4607669-f2d9-4ff1-8da0-9c0c19a3a570" containerName="revoke-keys" Jul 17 18:57:28.170859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170720 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4607669-f2d9-4ff1-8da0-9c0c19a3a570" containerName="revoke-keys" Jul 17 18:57:28.170859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170724 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4607669-f2d9-4ff1-8da0-9c0c19a3a570" containerName="revoke-keys" Jul 17 18:57:28.170859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170766 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="81ecb2fa-1902-4715-b873-11030369dca2" containerName="revoke-keys" Jul 17 18:57:28.170859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170773 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2644b523-29f2-497c-9bf8-5ca6e9294523" containerName="revoke-keys" Jul 17 18:57:28.170859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170779 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4607669-f2d9-4ff1-8da0-9c0c19a3a570" containerName="revoke-keys" Jul 17 18:57:28.170859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170784 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2644b523-29f2-497c-9bf8-5ca6e9294523" containerName="revoke-keys" Jul 17 18:57:28.170859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170789 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2644b523-29f2-497c-9bf8-5ca6e9294523" containerName="revoke-keys" Jul 17 18:57:28.170859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170795 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="81ecb2fa-1902-4715-b873-11030369dca2" containerName="revoke-keys" Jul 17 18:57:28.170859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170800 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4607669-f2d9-4ff1-8da0-9c0c19a3a570" containerName="revoke-keys" Jul 17 18:57:28.170859 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.170805 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4607669-f2d9-4ff1-8da0-9c0c19a3a570" containerName="revoke-keys" Jul 17 18:57:28.174789 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.174773 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m" Jul 17 18:57:28.177551 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.177531 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:57:28.177658 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.177531 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:57:28.181140 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.181052 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m"] Jul 17 18:57:28.262646 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.262614 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046a9907-a42b-4331-9aae-84c7aa4f2dad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m\" (UID: \"046a9907-a42b-4331-9aae-84c7aa4f2dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m" Jul 17 18:57:28.363847 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.363820 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046a9907-a42b-4331-9aae-84c7aa4f2dad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m\" (UID: \"046a9907-a42b-4331-9aae-84c7aa4f2dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m" Jul 17 18:57:28.364409 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.364389 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046a9907-a42b-4331-9aae-84c7aa4f2dad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m\" (UID: \"046a9907-a42b-4331-9aae-84c7aa4f2dad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m" Jul 17 18:57:28.486068 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.485959 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m" Jul 17 18:57:28.623357 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:28.623326 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m"] Jul 17 18:57:28.625919 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:57:28.625889 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046a9907_a42b_4331_9aae_84c7aa4f2dad.slice/crio-dbb6f2221d70607c1f923761c34b676d10d41c6688d6d97066f9ca5ad761b085 WatchSource:0}: Error finding container dbb6f2221d70607c1f923761c34b676d10d41c6688d6d97066f9ca5ad761b085: Status 404 returned error can't find the container with id dbb6f2221d70607c1f923761c34b676d10d41c6688d6d97066f9ca5ad761b085 Jul 17 18:57:29.582115 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:29.582077 2572 generic.go:356] "Generic (PLEG): container finished" podID="046a9907-a42b-4331-9aae-84c7aa4f2dad" containerID="c42dea2bde1144c7c65607a0571cadccf6b13b34fda42ee54193c19432b216e1" exitCode=6 Jul 17 18:57:29.582486 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:29.582125 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m" event={"ID":"046a9907-a42b-4331-9aae-84c7aa4f2dad","Type":"ContainerDied","Data":"c42dea2bde1144c7c65607a0571cadccf6b13b34fda42ee54193c19432b216e1"} Jul 17 18:57:29.582486 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:29.582172 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m" event={"ID":"046a9907-a42b-4331-9aae-84c7aa4f2dad","Type":"ContainerStarted","Data":"dbb6f2221d70607c1f923761c34b676d10d41c6688d6d97066f9ca5ad761b085"} Jul 17 18:57:29.582486 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:29.582374 2572 scope.go:117] "RemoveContainer" containerID="c42dea2bde1144c7c65607a0571cadccf6b13b34fda42ee54193c19432b216e1" Jul 17 18:57:30.589056 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:30.589003 2572 generic.go:356] "Generic (PLEG): container finished" podID="046a9907-a42b-4331-9aae-84c7aa4f2dad" containerID="c877702972a09d2f0f1a9f9723e5c8c18183183bd0b61dab0c781bce7c311ee3" exitCode=6 Jul 17 18:57:30.589556 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:30.589062 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m" event={"ID":"046a9907-a42b-4331-9aae-84c7aa4f2dad","Type":"ContainerDied","Data":"c877702972a09d2f0f1a9f9723e5c8c18183183bd0b61dab0c781bce7c311ee3"} Jul 17 18:57:30.589556 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:30.589113 2572 scope.go:117] "RemoveContainer" containerID="c42dea2bde1144c7c65607a0571cadccf6b13b34fda42ee54193c19432b216e1" Jul 17 18:57:30.589556 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:30.589340 2572 scope.go:117] "RemoveContainer" containerID="c877702972a09d2f0f1a9f9723e5c8c18183183bd0b61dab0c781bce7c311ee3" Jul 17 18:57:30.589728 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:57:30.589585 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m_odh-ai-gateway-infra(046a9907-a42b-4331-9aae-84c7aa4f2dad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m" podUID="046a9907-a42b-4331-9aae-84c7aa4f2dad" Jul 17 18:57:31.595035 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:31.594986 2572 scope.go:117] "RemoveContainer" containerID="c877702972a09d2f0f1a9f9723e5c8c18183183bd0b61dab0c781bce7c311ee3" Jul 17 18:57:31.595517 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:57:31.595236 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m_odh-ai-gateway-infra(046a9907-a42b-4331-9aae-84c7aa4f2dad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m" podUID="046a9907-a42b-4331-9aae-84c7aa4f2dad" Jul 17 18:57:32.361620 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:32.361585 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7278-139673086377392"] Jul 17 18:57:32.361911 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:32.361898 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2644b523-29f2-497c-9bf8-5ca6e9294523" containerName="revoke-keys" Jul 17 18:57:32.361981 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:32.361913 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2644b523-29f2-497c-9bf8-5ca6e9294523" containerName="revoke-keys" Jul 17 18:57:32.361981 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:32.361962 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="81ecb2fa-1902-4715-b873-11030369dca2" containerName="revoke-keys" Jul 17 18:57:32.365988 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:32.365972 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673086377392" Jul 17 18:57:32.370022 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:32.369988 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f4nxw\"" Jul 17 18:57:32.379802 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:32.379780 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7278-139673086377392"] Jul 17 18:57:32.498811 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:32.498780 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkp27\" (UniqueName: \"kubernetes.io/projected/01ba42e3-d453-41a0-b0bf-64e35b3aeb8b-kube-api-access-fkp27\") pod \"test-curl-7278-139673086377392\" (UID: \"01ba42e3-d453-41a0-b0bf-64e35b3aeb8b\") " pod="opendatahub/test-curl-7278-139673086377392" Jul 17 18:57:32.599716 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:32.599685 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkp27\" (UniqueName: \"kubernetes.io/projected/01ba42e3-d453-41a0-b0bf-64e35b3aeb8b-kube-api-access-fkp27\") pod \"test-curl-7278-139673086377392\" (UID: \"01ba42e3-d453-41a0-b0bf-64e35b3aeb8b\") " pod="opendatahub/test-curl-7278-139673086377392" Jul 17 18:57:32.609843 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:32.609812 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkp27\" (UniqueName: \"kubernetes.io/projected/01ba42e3-d453-41a0-b0bf-64e35b3aeb8b-kube-api-access-fkp27\") pod \"test-curl-7278-139673086377392\" (UID: \"01ba42e3-d453-41a0-b0bf-64e35b3aeb8b\") " pod="opendatahub/test-curl-7278-139673086377392" Jul 17 18:57:32.674793 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:32.674717 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673086377392" Jul 17 18:57:32.816412 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:32.816377 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7278-139673086377392"] Jul 17 18:57:32.818376 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:57:32.818345 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01ba42e3_d453_41a0_b0bf_64e35b3aeb8b.slice/crio-331fe546937edfe154db98fccbb5a924c7a13477cc9bdb4361ffe07840caf5dd WatchSource:0}: Error finding container 331fe546937edfe154db98fccbb5a924c7a13477cc9bdb4361ffe07840caf5dd: Status 404 returned error can't find the container with id 331fe546937edfe154db98fccbb5a924c7a13477cc9bdb4361ffe07840caf5dd Jul 17 18:57:32.819589 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:32.819574 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:57:33.605000 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:33.604972 2572 generic.go:356] "Generic (PLEG): container finished" podID="01ba42e3-d453-41a0-b0bf-64e35b3aeb8b" containerID="0be79f5b79abdbf2d15666cefede2b94840e5a50f0242cf119b1dadaa67c6468" exitCode=0 Jul 17 18:57:33.605362 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:33.605052 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7278-139673086377392" event={"ID":"01ba42e3-d453-41a0-b0bf-64e35b3aeb8b","Type":"ContainerDied","Data":"0be79f5b79abdbf2d15666cefede2b94840e5a50f0242cf119b1dadaa67c6468"} Jul 17 18:57:33.605362 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:33.605093 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7278-139673086377392" event={"ID":"01ba42e3-d453-41a0-b0bf-64e35b3aeb8b","Type":"ContainerStarted","Data":"331fe546937edfe154db98fccbb5a924c7a13477cc9bdb4361ffe07840caf5dd"} Jul 17 18:57:34.738251 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:34.738226 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673086377392" Jul 17 18:57:34.781281 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:34.781250 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7278-139673086377392_01ba42e3-d453-41a0-b0bf-64e35b3aeb8b/test-curl-7278-139673086377392/0.log" Jul 17 18:57:34.809374 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:34.809343 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7278-139673086377392"] Jul 17 18:57:34.814240 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:34.814217 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7278-139673086377392"] Jul 17 18:57:34.820910 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:34.820894 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkp27\" (UniqueName: \"kubernetes.io/projected/01ba42e3-d453-41a0-b0bf-64e35b3aeb8b-kube-api-access-fkp27\") pod \"01ba42e3-d453-41a0-b0bf-64e35b3aeb8b\" (UID: \"01ba42e3-d453-41a0-b0bf-64e35b3aeb8b\") " Jul 17 18:57:34.823141 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:34.823120 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ba42e3-d453-41a0-b0bf-64e35b3aeb8b-kube-api-access-fkp27" (OuterVolumeSpecName: "kube-api-access-fkp27") pod "01ba42e3-d453-41a0-b0bf-64e35b3aeb8b" (UID: "01ba42e3-d453-41a0-b0bf-64e35b3aeb8b"). InnerVolumeSpecName "kube-api-access-fkp27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:57:34.922415 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:34.922331 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkp27\" (UniqueName: \"kubernetes.io/projected/01ba42e3-d453-41a0-b0bf-64e35b3aeb8b-kube-api-access-fkp27\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:57:34.938508 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:34.938477 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7278-139673085257264"] Jul 17 18:57:34.938804 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:34.938792 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01ba42e3-d453-41a0-b0bf-64e35b3aeb8b" containerName="test-curl-7278-139673086377392" Jul 17 18:57:34.938849 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:34.938806 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="01ba42e3-d453-41a0-b0bf-64e35b3aeb8b" containerName="test-curl-7278-139673086377392" Jul 17 18:57:34.938884 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:34.938863 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="01ba42e3-d453-41a0-b0bf-64e35b3aeb8b" containerName="test-curl-7278-139673086377392" Jul 17 18:57:34.941717 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:34.941702 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673085257264" Jul 17 18:57:34.946867 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:34.946838 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7278-139673085257264"] Jul 17 18:57:35.023167 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:35.023136 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnh2x\" (UniqueName: \"kubernetes.io/projected/ff74ff7b-14f2-4878-b250-2a6603d12879-kube-api-access-bnh2x\") pod \"test-curl-7278-139673085257264\" (UID: \"ff74ff7b-14f2-4878-b250-2a6603d12879\") " pod="opendatahub/test-curl-7278-139673085257264" Jul 17 18:57:35.124545 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:35.124513 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnh2x\" (UniqueName: \"kubernetes.io/projected/ff74ff7b-14f2-4878-b250-2a6603d12879-kube-api-access-bnh2x\") pod \"test-curl-7278-139673085257264\" (UID: \"ff74ff7b-14f2-4878-b250-2a6603d12879\") " pod="opendatahub/test-curl-7278-139673085257264" Jul 17 18:57:35.134116 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:35.134095 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnh2x\" (UniqueName: \"kubernetes.io/projected/ff74ff7b-14f2-4878-b250-2a6603d12879-kube-api-access-bnh2x\") pod \"test-curl-7278-139673085257264\" (UID: \"ff74ff7b-14f2-4878-b250-2a6603d12879\") " pod="opendatahub/test-curl-7278-139673085257264" Jul 17 18:57:35.224590 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:35.224506 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01ba42e3-d453-41a0-b0bf-64e35b3aeb8b" path="/var/lib/kubelet/pods/01ba42e3-d453-41a0-b0bf-64e35b3aeb8b/volumes" Jul 17 18:57:35.251897 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:35.251865 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673085257264" Jul 17 18:57:35.378858 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:35.378835 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7278-139673085257264"] Jul 17 18:57:35.380510 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:57:35.380491 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff74ff7b_14f2_4878_b250_2a6603d12879.slice/crio-0e1d72c462a6fdaa757904917b54d4552fcd7087c936e35f58c14a129f7bfae5 WatchSource:0}: Error finding container 0e1d72c462a6fdaa757904917b54d4552fcd7087c936e35f58c14a129f7bfae5: Status 404 returned error can't find the container with id 0e1d72c462a6fdaa757904917b54d4552fcd7087c936e35f58c14a129f7bfae5 Jul 17 18:57:35.614716 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:35.614680 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7278-139673085257264" event={"ID":"ff74ff7b-14f2-4878-b250-2a6603d12879","Type":"ContainerStarted","Data":"0e1d72c462a6fdaa757904917b54d4552fcd7087c936e35f58c14a129f7bfae5"} Jul 17 18:57:35.615702 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:35.615682 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673086377392" Jul 17 18:57:35.615702 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:35.615698 2572 scope.go:117] "RemoveContainer" containerID="0be79f5b79abdbf2d15666cefede2b94840e5a50f0242cf119b1dadaa67c6468" Jul 17 18:57:36.621910 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:36.621876 2572 generic.go:356] "Generic (PLEG): container finished" podID="ff74ff7b-14f2-4878-b250-2a6603d12879" containerID="a53a0a47bcd5f9f2ed928c1f4b039eaa1ae4b5cb3f35fa2248bd1b0303f8c1e0" exitCode=0 Jul 17 18:57:36.622380 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:36.621958 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7278-139673085257264" event={"ID":"ff74ff7b-14f2-4878-b250-2a6603d12879","Type":"ContainerDied","Data":"a53a0a47bcd5f9f2ed928c1f4b039eaa1ae4b5cb3f35fa2248bd1b0303f8c1e0"} Jul 17 18:57:37.753869 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:37.753840 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673085257264" Jul 17 18:57:37.779225 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:37.779197 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7278-139673085257264_ff74ff7b-14f2-4878-b250-2a6603d12879/test-curl-7278-139673085257264/0.log" Jul 17 18:57:37.802464 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:37.802440 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7278-139673085257264"] Jul 17 18:57:37.812796 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:37.812768 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7278-139673085257264"] Jul 17 18:57:37.844559 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:37.844540 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnh2x\" (UniqueName: \"kubernetes.io/projected/ff74ff7b-14f2-4878-b250-2a6603d12879-kube-api-access-bnh2x\") pod \"ff74ff7b-14f2-4878-b250-2a6603d12879\" (UID: \"ff74ff7b-14f2-4878-b250-2a6603d12879\") " Jul 17 18:57:37.846591 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:37.846567 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff74ff7b-14f2-4878-b250-2a6603d12879-kube-api-access-bnh2x" (OuterVolumeSpecName: "kube-api-access-bnh2x") pod "ff74ff7b-14f2-4878-b250-2a6603d12879" (UID: "ff74ff7b-14f2-4878-b250-2a6603d12879"). InnerVolumeSpecName "kube-api-access-bnh2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:57:37.945151 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:37.945077 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnh2x\" (UniqueName: \"kubernetes.io/projected/ff74ff7b-14f2-4878-b250-2a6603d12879-kube-api-access-bnh2x\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:57:37.987736 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:37.987710 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7278-139673085020464"] Jul 17 18:57:37.988065 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:37.988050 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff74ff7b-14f2-4878-b250-2a6603d12879" containerName="test-curl-7278-139673085257264" Jul 17 18:57:37.988118 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:37.988066 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff74ff7b-14f2-4878-b250-2a6603d12879" containerName="test-curl-7278-139673085257264" Jul 17 18:57:37.988154 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:37.988130 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff74ff7b-14f2-4878-b250-2a6603d12879" containerName="test-curl-7278-139673085257264" Jul 17 18:57:37.991035 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:37.991001 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673085020464" Jul 17 18:57:37.999567 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:37.999545 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7278-139673085020464"] Jul 17 18:57:38.045919 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:38.045887 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rbvt\" (UniqueName: \"kubernetes.io/projected/b483357c-efc1-4bce-b10f-2a68401b870b-kube-api-access-2rbvt\") pod \"test-curl-7278-139673085020464\" (UID: \"b483357c-efc1-4bce-b10f-2a68401b870b\") " pod="opendatahub/test-curl-7278-139673085020464" Jul 17 18:57:38.147383 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:38.147357 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rbvt\" (UniqueName: \"kubernetes.io/projected/b483357c-efc1-4bce-b10f-2a68401b870b-kube-api-access-2rbvt\") pod \"test-curl-7278-139673085020464\" (UID: \"b483357c-efc1-4bce-b10f-2a68401b870b\") " pod="opendatahub/test-curl-7278-139673085020464" Jul 17 18:57:38.170721 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:38.170694 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rbvt\" (UniqueName: \"kubernetes.io/projected/b483357c-efc1-4bce-b10f-2a68401b870b-kube-api-access-2rbvt\") pod \"test-curl-7278-139673085020464\" (UID: \"b483357c-efc1-4bce-b10f-2a68401b870b\") " pod="opendatahub/test-curl-7278-139673085020464" Jul 17 18:57:38.300792 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:38.300762 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673085020464" Jul 17 18:57:38.436687 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:38.436584 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7278-139673085020464"] Jul 17 18:57:38.439176 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:57:38.439152 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb483357c_efc1_4bce_b10f_2a68401b870b.slice/crio-6f6e9aa84494edfc881c54df9ba03abc212e26869e757f4386bd8a12611cc674 WatchSource:0}: Error finding container 6f6e9aa84494edfc881c54df9ba03abc212e26869e757f4386bd8a12611cc674: Status 404 returned error can't find the container with id 6f6e9aa84494edfc881c54df9ba03abc212e26869e757f4386bd8a12611cc674 Jul 17 18:57:38.633169 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:38.633082 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673085257264" Jul 17 18:57:38.633169 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:38.633087 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e1d72c462a6fdaa757904917b54d4552fcd7087c936e35f58c14a129f7bfae5" Jul 17 18:57:38.634304 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:38.634274 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7278-139673085020464" event={"ID":"b483357c-efc1-4bce-b10f-2a68401b870b","Type":"ContainerStarted","Data":"6f6e9aa84494edfc881c54df9ba03abc212e26869e757f4386bd8a12611cc674"} Jul 17 18:57:38.643889 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:57:38.643864 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7278-139673085257264\" is forbidden: User \"system:node:ip-10-0-132-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-62.ec2.internal' and this object" podUID="ff74ff7b-14f2-4878-b250-2a6603d12879" pod="opendatahub/test-curl-7278-139673085257264" Jul 17 18:57:39.225330 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:39.225298 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff74ff7b-14f2-4878-b250-2a6603d12879" path="/var/lib/kubelet/pods/ff74ff7b-14f2-4878-b250-2a6603d12879/volumes" Jul 17 18:57:39.640329 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:39.640291 2572 generic.go:356] "Generic (PLEG): container finished" podID="b483357c-efc1-4bce-b10f-2a68401b870b" containerID="ad6e3f0a1a476a19e42603220887c13c92bf77cb618397f835fd09d7b7003826" exitCode=0 Jul 17 18:57:39.640504 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:39.640337 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7278-139673085020464" event={"ID":"b483357c-efc1-4bce-b10f-2a68401b870b","Type":"ContainerDied","Data":"ad6e3f0a1a476a19e42603220887c13c92bf77cb618397f835fd09d7b7003826"} Jul 17 18:57:40.777887 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:40.777862 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673085020464" Jul 17 18:57:40.798727 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:40.798704 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7278-139673085020464_b483357c-efc1-4bce-b10f-2a68401b870b/test-curl-7278-139673085020464/0.log" Jul 17 18:57:40.823729 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:40.823704 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7278-139673085020464"] Jul 17 18:57:40.829104 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:40.829078 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7278-139673085020464"] Jul 17 18:57:40.870672 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:40.870644 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rbvt\" (UniqueName: \"kubernetes.io/projected/b483357c-efc1-4bce-b10f-2a68401b870b-kube-api-access-2rbvt\") pod \"b483357c-efc1-4bce-b10f-2a68401b870b\" (UID: \"b483357c-efc1-4bce-b10f-2a68401b870b\") " Jul 17 18:57:40.872673 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:40.872654 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b483357c-efc1-4bce-b10f-2a68401b870b-kube-api-access-2rbvt" (OuterVolumeSpecName: "kube-api-access-2rbvt") pod "b483357c-efc1-4bce-b10f-2a68401b870b" (UID: "b483357c-efc1-4bce-b10f-2a68401b870b"). InnerVolumeSpecName "kube-api-access-2rbvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:57:40.960718 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:40.960645 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7278-139673085020464"] Jul 17 18:57:40.961057 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:40.961041 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b483357c-efc1-4bce-b10f-2a68401b870b" containerName="test-curl-7278-139673085020464" Jul 17 18:57:40.961133 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:40.961060 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b483357c-efc1-4bce-b10f-2a68401b870b" containerName="test-curl-7278-139673085020464" Jul 17 18:57:40.961199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:40.961142 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b483357c-efc1-4bce-b10f-2a68401b870b" containerName="test-curl-7278-139673085020464" Jul 17 18:57:40.964115 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:40.964097 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673085020464" Jul 17 18:57:40.970948 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:40.970799 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7278-139673085020464"] Jul 17 18:57:40.971573 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:40.971506 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rbvt\" (UniqueName: \"kubernetes.io/projected/b483357c-efc1-4bce-b10f-2a68401b870b-kube-api-access-2rbvt\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:57:41.072815 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:41.072783 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4859\" (UniqueName: \"kubernetes.io/projected/17a1e752-bdf0-47f7-9450-b0d92bb4972d-kube-api-access-q4859\") pod \"test-curl-7278-139673085020464\" (UID: \"17a1e752-bdf0-47f7-9450-b0d92bb4972d\") " pod="opendatahub/test-curl-7278-139673085020464" Jul 17 18:57:41.174085 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:41.174052 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4859\" (UniqueName: \"kubernetes.io/projected/17a1e752-bdf0-47f7-9450-b0d92bb4972d-kube-api-access-q4859\") pod \"test-curl-7278-139673085020464\" (UID: \"17a1e752-bdf0-47f7-9450-b0d92bb4972d\") " pod="opendatahub/test-curl-7278-139673085020464" Jul 17 18:57:41.181656 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:41.181628 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4859\" (UniqueName: \"kubernetes.io/projected/17a1e752-bdf0-47f7-9450-b0d92bb4972d-kube-api-access-q4859\") pod \"test-curl-7278-139673085020464\" (UID: \"17a1e752-bdf0-47f7-9450-b0d92bb4972d\") " pod="opendatahub/test-curl-7278-139673085020464" Jul 17 18:57:41.225123 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:41.225054 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b483357c-efc1-4bce-b10f-2a68401b870b" path="/var/lib/kubelet/pods/b483357c-efc1-4bce-b10f-2a68401b870b/volumes" Jul 17 18:57:41.274828 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:41.274797 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673085020464" Jul 17 18:57:41.397563 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:41.397536 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7278-139673085020464"] Jul 17 18:57:41.399947 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:57:41.399925 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17a1e752_bdf0_47f7_9450_b0d92bb4972d.slice/crio-cf473e92958f7d37bd7ea0bf178490876a74b8a4325183afbc31a1b8c444111b WatchSource:0}: Error finding container cf473e92958f7d37bd7ea0bf178490876a74b8a4325183afbc31a1b8c444111b: Status 404 returned error can't find the container with id cf473e92958f7d37bd7ea0bf178490876a74b8a4325183afbc31a1b8c444111b Jul 17 18:57:41.650811 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:41.650770 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7278-139673085020464" event={"ID":"17a1e752-bdf0-47f7-9450-b0d92bb4972d","Type":"ContainerStarted","Data":"cf473e92958f7d37bd7ea0bf178490876a74b8a4325183afbc31a1b8c444111b"} Jul 17 18:57:41.652137 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:41.652110 2572 scope.go:117] "RemoveContainer" containerID="ad6e3f0a1a476a19e42603220887c13c92bf77cb618397f835fd09d7b7003826" Jul 17 18:57:41.652283 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:41.652111 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673085020464" Jul 17 18:57:42.657462 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:42.657426 2572 generic.go:356] "Generic (PLEG): container finished" podID="17a1e752-bdf0-47f7-9450-b0d92bb4972d" containerID="fb5b5c8875cd63c97819054913c7fd9c86316e63962fa26e43af4a32a6ab67ba" exitCode=0 Jul 17 18:57:42.657890 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:42.657512 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7278-139673085020464" event={"ID":"17a1e752-bdf0-47f7-9450-b0d92bb4972d","Type":"ContainerDied","Data":"fb5b5c8875cd63c97819054913c7fd9c86316e63962fa26e43af4a32a6ab67ba"} Jul 17 18:57:43.788530 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:43.788505 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673085020464" Jul 17 18:57:43.817427 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:43.817404 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7278-139673085020464_17a1e752-bdf0-47f7-9450-b0d92bb4972d/test-curl-7278-139673085020464/0.log" Jul 17 18:57:43.849197 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:43.849166 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7278-139673085020464"] Jul 17 18:57:43.853430 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:43.853408 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7278-139673085020464"] Jul 17 18:57:43.896909 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:43.896890 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4859\" (UniqueName: \"kubernetes.io/projected/17a1e752-bdf0-47f7-9450-b0d92bb4972d-kube-api-access-q4859\") pod \"17a1e752-bdf0-47f7-9450-b0d92bb4972d\" (UID: \"17a1e752-bdf0-47f7-9450-b0d92bb4972d\") " Jul 17 18:57:43.898891 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:43.898862 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a1e752-bdf0-47f7-9450-b0d92bb4972d-kube-api-access-q4859" (OuterVolumeSpecName: "kube-api-access-q4859") pod "17a1e752-bdf0-47f7-9450-b0d92bb4972d" (UID: "17a1e752-bdf0-47f7-9450-b0d92bb4972d"). InnerVolumeSpecName "kube-api-access-q4859". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:57:43.983199 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:43.983133 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7278-139673085021472"] Jul 17 18:57:43.983536 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:43.983519 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a1e752-bdf0-47f7-9450-b0d92bb4972d" containerName="test-curl-7278-139673085020464" Jul 17 18:57:43.983585 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:43.983539 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a1e752-bdf0-47f7-9450-b0d92bb4972d" containerName="test-curl-7278-139673085020464" Jul 17 18:57:43.983652 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:43.983640 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a1e752-bdf0-47f7-9450-b0d92bb4972d" containerName="test-curl-7278-139673085020464" Jul 17 18:57:43.986579 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:43.986564 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673085021472" Jul 17 18:57:43.994665 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:43.994637 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7278-139673085021472"] Jul 17 18:57:43.997943 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:43.997920 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4859\" (UniqueName: \"kubernetes.io/projected/17a1e752-bdf0-47f7-9450-b0d92bb4972d-kube-api-access-q4859\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:57:44.098339 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:44.098303 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-299xn\" (UniqueName: \"kubernetes.io/projected/87e7a9ef-96af-4a71-89c8-4df7bf065597-kube-api-access-299xn\") pod \"test-curl-7278-139673085021472\" (UID: \"87e7a9ef-96af-4a71-89c8-4df7bf065597\") " pod="opendatahub/test-curl-7278-139673085021472" Jul 17 18:57:44.199034 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:44.198979 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-299xn\" (UniqueName: \"kubernetes.io/projected/87e7a9ef-96af-4a71-89c8-4df7bf065597-kube-api-access-299xn\") pod \"test-curl-7278-139673085021472\" (UID: \"87e7a9ef-96af-4a71-89c8-4df7bf065597\") " pod="opendatahub/test-curl-7278-139673085021472" Jul 17 18:57:44.209938 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:44.209913 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-299xn\" (UniqueName: \"kubernetes.io/projected/87e7a9ef-96af-4a71-89c8-4df7bf065597-kube-api-access-299xn\") pod \"test-curl-7278-139673085021472\" (UID: \"87e7a9ef-96af-4a71-89c8-4df7bf065597\") " pod="opendatahub/test-curl-7278-139673085021472" Jul 17 18:57:44.220122 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:44.220104 2572 scope.go:117] "RemoveContainer" containerID="c877702972a09d2f0f1a9f9723e5c8c18183183bd0b61dab0c781bce7c311ee3" Jul 17 18:57:44.296188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:44.296148 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673085021472" Jul 17 18:57:44.427607 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:44.427583 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7278-139673085021472"] Jul 17 18:57:44.428275 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:57:44.428249 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87e7a9ef_96af_4a71_89c8_4df7bf065597.slice/crio-45e53f50cc2be817bafac2b9808017c61eb76e258bc251fa6c8d2a24691c86d4 WatchSource:0}: Error finding container 45e53f50cc2be817bafac2b9808017c61eb76e258bc251fa6c8d2a24691c86d4: Status 404 returned error can't find the container with id 45e53f50cc2be817bafac2b9808017c61eb76e258bc251fa6c8d2a24691c86d4 Jul 17 18:57:44.668544 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:44.668503 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m" event={"ID":"046a9907-a42b-4331-9aae-84c7aa4f2dad","Type":"ContainerStarted","Data":"deb4c65c72f8b37cd5c4879c78296b9c3b90f7ce480e85b53e92f8fafd3d3f94"} Jul 17 18:57:44.669647 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:44.669623 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7278-139673085021472" event={"ID":"87e7a9ef-96af-4a71-89c8-4df7bf065597","Type":"ContainerStarted","Data":"45e53f50cc2be817bafac2b9808017c61eb76e258bc251fa6c8d2a24691c86d4"} Jul 17 18:57:44.670711 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:44.670696 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf473e92958f7d37bd7ea0bf178490876a74b8a4325183afbc31a1b8c444111b" Jul 17 18:57:44.670790 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:44.670712 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673085020464" Jul 17 18:57:44.700291 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:44.700246 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m" podStartSLOduration=16.700232859 podStartE2EDuration="16.700232859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:57:44.699269777 +0000 UTC m=+3660.108697374" watchObservedRunningTime="2026-07-17 18:57:44.700232859 +0000 UTC m=+3660.109660464" Jul 17 18:57:44.706389 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:57:44.706364 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7278-139673085020464\" is forbidden: User \"system:node:ip-10-0-132-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-62.ec2.internal' and this object" podUID="17a1e752-bdf0-47f7-9450-b0d92bb4972d" pod="opendatahub/test-curl-7278-139673085020464" Jul 17 18:57:45.226476 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:57:45.226384 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7278-139673085020464\" is forbidden: User \"system:node:ip-10-0-132-62.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-62.ec2.internal' and this object" podUID="17a1e752-bdf0-47f7-9450-b0d92bb4972d" pod="opendatahub/test-curl-7278-139673085020464" Jul 17 18:57:45.229260 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:45.229236 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17a1e752-bdf0-47f7-9450-b0d92bb4972d" path="/var/lib/kubelet/pods/17a1e752-bdf0-47f7-9450-b0d92bb4972d/volumes" Jul 17 18:57:45.677222 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:45.677191 2572 generic.go:356] "Generic (PLEG): container finished" podID="046a9907-a42b-4331-9aae-84c7aa4f2dad" containerID="deb4c65c72f8b37cd5c4879c78296b9c3b90f7ce480e85b53e92f8fafd3d3f94" exitCode=6 Jul 17 18:57:45.677412 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:45.677263 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m" event={"ID":"046a9907-a42b-4331-9aae-84c7aa4f2dad","Type":"ContainerDied","Data":"deb4c65c72f8b37cd5c4879c78296b9c3b90f7ce480e85b53e92f8fafd3d3f94"} Jul 17 18:57:45.677412 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:45.677318 2572 scope.go:117] "RemoveContainer" containerID="c877702972a09d2f0f1a9f9723e5c8c18183183bd0b61dab0c781bce7c311ee3" Jul 17 18:57:45.677630 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:45.677607 2572 scope.go:117] "RemoveContainer" containerID="deb4c65c72f8b37cd5c4879c78296b9c3b90f7ce480e85b53e92f8fafd3d3f94" Jul 17 18:57:45.677866 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:57:45.677845 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m_odh-ai-gateway-infra(046a9907-a42b-4331-9aae-84c7aa4f2dad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m" podUID="046a9907-a42b-4331-9aae-84c7aa4f2dad" Jul 17 18:57:45.678785 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:45.678761 2572 generic.go:356] "Generic (PLEG): container finished" podID="87e7a9ef-96af-4a71-89c8-4df7bf065597" containerID="113f03946982f56a28abeb6164fbf15913ab9797621793e1af382daf99b2402f" exitCode=0 Jul 17 18:57:45.678901 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:45.678823 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7278-139673085021472" event={"ID":"87e7a9ef-96af-4a71-89c8-4df7bf065597","Type":"ContainerDied","Data":"113f03946982f56a28abeb6164fbf15913ab9797621793e1af382daf99b2402f"} Jul 17 18:57:45.746139 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:45.746116 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m"] Jul 17 18:57:46.081372 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:46.081341 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k"] Jul 17 18:57:46.086003 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:46.085987 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k" Jul 17 18:57:46.105679 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:46.105648 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k"] Jul 17 18:57:46.217637 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:46.217605 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e5601d-4d66-44cd-a8df-f38885281dff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k\" (UID: \"b1e5601d-4d66-44cd-a8df-f38885281dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k" Jul 17 18:57:46.318952 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:46.318922 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e5601d-4d66-44cd-a8df-f38885281dff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k\" (UID: \"b1e5601d-4d66-44cd-a8df-f38885281dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k" Jul 17 18:57:46.319546 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:46.319528 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e5601d-4d66-44cd-a8df-f38885281dff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k\" (UID: \"b1e5601d-4d66-44cd-a8df-f38885281dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k" Jul 17 18:57:46.396073 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:46.395980 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k" Jul 17 18:57:46.544307 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:46.544274 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k"] Jul 17 18:57:46.548122 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:57:46.548093 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e5601d_4d66_44cd_a8df_f38885281dff.slice/crio-ca0c6ce26c32e768dc012b9cc7c2bd685a7aeabd93ea6b9637b90592d4529d28 WatchSource:0}: Error finding container ca0c6ce26c32e768dc012b9cc7c2bd685a7aeabd93ea6b9637b90592d4529d28: Status 404 returned error can't find the container with id ca0c6ce26c32e768dc012b9cc7c2bd685a7aeabd93ea6b9637b90592d4529d28 Jul 17 18:57:46.684562 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:46.684481 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k" event={"ID":"b1e5601d-4d66-44cd-a8df-f38885281dff","Type":"ContainerStarted","Data":"ca0c6ce26c32e768dc012b9cc7c2bd685a7aeabd93ea6b9637b90592d4529d28"} Jul 17 18:57:46.819379 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:46.819356 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673085021472" Jul 17 18:57:46.834332 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:46.834304 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m" Jul 17 18:57:46.848000 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:46.847974 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7278-139673085021472_87e7a9ef-96af-4a71-89c8-4df7bf065597/test-curl-7278-139673085021472/0.log" Jul 17 18:57:46.870876 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:46.870852 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7278-139673085021472"] Jul 17 18:57:46.875360 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:46.875336 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7278-139673085021472"] Jul 17 18:57:46.924539 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:46.924513 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-299xn\" (UniqueName: \"kubernetes.io/projected/87e7a9ef-96af-4a71-89c8-4df7bf065597-kube-api-access-299xn\") pod \"87e7a9ef-96af-4a71-89c8-4df7bf065597\" (UID: \"87e7a9ef-96af-4a71-89c8-4df7bf065597\") " Jul 17 18:57:46.924654 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:46.924552 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046a9907-a42b-4331-9aae-84c7aa4f2dad-maas-api-service-ca\") pod \"046a9907-a42b-4331-9aae-84c7aa4f2dad\" (UID: \"046a9907-a42b-4331-9aae-84c7aa4f2dad\") " Jul 17 18:57:46.924914 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:46.924891 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046a9907-a42b-4331-9aae-84c7aa4f2dad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "046a9907-a42b-4331-9aae-84c7aa4f2dad" (UID: "046a9907-a42b-4331-9aae-84c7aa4f2dad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:57:46.926471 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:46.926453 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e7a9ef-96af-4a71-89c8-4df7bf065597-kube-api-access-299xn" (OuterVolumeSpecName: "kube-api-access-299xn") pod "87e7a9ef-96af-4a71-89c8-4df7bf065597" (UID: "87e7a9ef-96af-4a71-89c8-4df7bf065597"). InnerVolumeSpecName "kube-api-access-299xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:57:47.025814 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:47.025790 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-299xn\" (UniqueName: \"kubernetes.io/projected/87e7a9ef-96af-4a71-89c8-4df7bf065597-kube-api-access-299xn\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:57:47.025929 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:47.025816 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046a9907-a42b-4331-9aae-84c7aa4f2dad-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:57:47.224997 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:47.224963 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87e7a9ef-96af-4a71-89c8-4df7bf065597" path="/var/lib/kubelet/pods/87e7a9ef-96af-4a71-89c8-4df7bf065597/volumes" Jul 17 18:57:47.694702 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:47.694618 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m" Jul 17 18:57:47.694702 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:47.694634 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m" event={"ID":"046a9907-a42b-4331-9aae-84c7aa4f2dad","Type":"ContainerDied","Data":"dbb6f2221d70607c1f923761c34b676d10d41c6688d6d97066f9ca5ad761b085"} Jul 17 18:57:47.694702 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:47.694693 2572 scope.go:117] "RemoveContainer" containerID="deb4c65c72f8b37cd5c4879c78296b9c3b90f7ce480e85b53e92f8fafd3d3f94" Jul 17 18:57:47.696368 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:47.696342 2572 generic.go:356] "Generic (PLEG): container finished" podID="b1e5601d-4d66-44cd-a8df-f38885281dff" containerID="180db4ec0318c49b22b15b3d53e597130c80693606b7c7e7166d32c37ae5016b" exitCode=6 Jul 17 18:57:47.696521 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:47.696411 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k" event={"ID":"b1e5601d-4d66-44cd-a8df-f38885281dff","Type":"ContainerDied","Data":"180db4ec0318c49b22b15b3d53e597130c80693606b7c7e7166d32c37ae5016b"} Jul 17 18:57:47.696678 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:47.696617 2572 scope.go:117] "RemoveContainer" containerID="180db4ec0318c49b22b15b3d53e597130c80693606b7c7e7166d32c37ae5016b" Jul 17 18:57:47.697863 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:47.697846 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-139673085021472" Jul 17 18:57:47.703945 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:47.703922 2572 scope.go:117] "RemoveContainer" containerID="113f03946982f56a28abeb6164fbf15913ab9797621793e1af382daf99b2402f" Jul 17 18:57:47.719004 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:47.718978 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m"] Jul 17 18:57:47.725761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:47.725724 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-pk95m"] Jul 17 18:57:48.704437 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:48.704351 2572 generic.go:356] "Generic (PLEG): container finished" podID="b1e5601d-4d66-44cd-a8df-f38885281dff" containerID="47e008aba1d897f8f90ae337f616787b2ee1f549d86c7b52310407adb5c99df9" exitCode=6 Jul 17 18:57:48.704865 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:48.704436 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k" event={"ID":"b1e5601d-4d66-44cd-a8df-f38885281dff","Type":"ContainerDied","Data":"47e008aba1d897f8f90ae337f616787b2ee1f549d86c7b52310407adb5c99df9"} Jul 17 18:57:48.704865 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:48.704498 2572 scope.go:117] "RemoveContainer" containerID="180db4ec0318c49b22b15b3d53e597130c80693606b7c7e7166d32c37ae5016b" Jul 17 18:57:48.704865 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:48.704617 2572 scope.go:117] "RemoveContainer" containerID="47e008aba1d897f8f90ae337f616787b2ee1f549d86c7b52310407adb5c99df9" Jul 17 18:57:48.704865 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:57:48.704826 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k_odh-ai-gateway-infra(b1e5601d-4d66-44cd-a8df-f38885281dff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k" podUID="b1e5601d-4d66-44cd-a8df-f38885281dff" Jul 17 18:57:49.224992 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:49.224962 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="046a9907-a42b-4331-9aae-84c7aa4f2dad" path="/var/lib/kubelet/pods/046a9907-a42b-4331-9aae-84c7aa4f2dad/volumes" Jul 17 18:57:49.712930 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:49.712902 2572 scope.go:117] "RemoveContainer" containerID="47e008aba1d897f8f90ae337f616787b2ee1f549d86c7b52310407adb5c99df9" Jul 17 18:57:49.713422 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:57:49.713161 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k_odh-ai-gateway-infra(b1e5601d-4d66-44cd-a8df-f38885281dff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k" podUID="b1e5601d-4d66-44cd-a8df-f38885281dff" Jul 17 18:57:50.090620 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.090585 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7d6f90-25tsp"] Jul 17 18:57:50.090883 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.090871 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046a9907-a42b-4331-9aae-84c7aa4f2dad" containerName="revoke-keys" Jul 17 18:57:50.090928 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.090885 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="046a9907-a42b-4331-9aae-84c7aa4f2dad" containerName="revoke-keys" Jul 17 18:57:50.090928 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.090894 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046a9907-a42b-4331-9aae-84c7aa4f2dad" containerName="revoke-keys" Jul 17 18:57:50.090928 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.090900 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="046a9907-a42b-4331-9aae-84c7aa4f2dad" containerName="revoke-keys" Jul 17 18:57:50.090928 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.090909 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87e7a9ef-96af-4a71-89c8-4df7bf065597" containerName="test-curl-7278-139673085021472" Jul 17 18:57:50.090928 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.090915 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="87e7a9ef-96af-4a71-89c8-4df7bf065597" containerName="test-curl-7278-139673085021472" Jul 17 18:57:50.091122 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.090972 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="046a9907-a42b-4331-9aae-84c7aa4f2dad" containerName="revoke-keys" Jul 17 18:57:50.091122 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.090979 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="046a9907-a42b-4331-9aae-84c7aa4f2dad" containerName="revoke-keys" Jul 17 18:57:50.091122 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.090985 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="87e7a9ef-96af-4a71-89c8-4df7bf065597" containerName="test-curl-7278-139673085021472" Jul 17 18:57:50.091122 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.090991 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="046a9907-a42b-4331-9aae-84c7aa4f2dad" containerName="revoke-keys" Jul 17 18:57:50.095299 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.095280 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7d6f90-25tsp" Jul 17 18:57:50.107240 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.107213 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7d6f90-25tsp"] Jul 17 18:57:50.253407 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.253370 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a73792-4695-4e1d-a4c9-17d09ae1e05e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-7d6f90-25tsp\" (UID: \"64a73792-4695-4e1d-a4c9-17d09ae1e05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7d6f90-25tsp" Jul 17 18:57:50.354371 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.354293 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a73792-4695-4e1d-a4c9-17d09ae1e05e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-7d6f90-25tsp\" (UID: \"64a73792-4695-4e1d-a4c9-17d09ae1e05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7d6f90-25tsp" Jul 17 18:57:50.354864 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.354846 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a73792-4695-4e1d-a4c9-17d09ae1e05e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-7d6f90-25tsp\" (UID: \"64a73792-4695-4e1d-a4c9-17d09ae1e05e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7d6f90-25tsp" Jul 17 18:57:50.405377 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.405357 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7d6f90-25tsp" Jul 17 18:57:50.532347 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.532315 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7d6f90-25tsp"] Jul 17 18:57:50.535078 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:57:50.535046 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a73792_4695_4e1d_a4c9_17d09ae1e05e.slice/crio-ea72770efe430b9ce0bba0b48f0cf094364562a449e062e1f1327a3164ec840c WatchSource:0}: Error finding container ea72770efe430b9ce0bba0b48f0cf094364562a449e062e1f1327a3164ec840c: Status 404 returned error can't find the container with id ea72770efe430b9ce0bba0b48f0cf094364562a449e062e1f1327a3164ec840c Jul 17 18:57:50.718608 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:50.718531 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7d6f90-25tsp" event={"ID":"64a73792-4695-4e1d-a4c9-17d09ae1e05e","Type":"ContainerStarted","Data":"ea72770efe430b9ce0bba0b48f0cf094364562a449e062e1f1327a3164ec840c"} Jul 17 18:57:51.724893 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:51.724862 2572 generic.go:356] "Generic (PLEG): container finished" podID="64a73792-4695-4e1d-a4c9-17d09ae1e05e" containerID="34276324edc0d65d6de6965775750e5173f774de8c2accea277b63fb66662ec8" exitCode=0 Jul 17 18:57:51.725337 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:51.724911 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7d6f90-25tsp" event={"ID":"64a73792-4695-4e1d-a4c9-17d09ae1e05e","Type":"ContainerDied","Data":"34276324edc0d65d6de6965775750e5173f774de8c2accea277b63fb66662ec8"} Jul 17 18:57:52.863153 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:52.863127 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7d6f90-25tsp" Jul 17 18:57:52.978394 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:52.978363 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a73792-4695-4e1d-a4c9-17d09ae1e05e-maas-api-service-ca\") pod \"64a73792-4695-4e1d-a4c9-17d09ae1e05e\" (UID: \"64a73792-4695-4e1d-a4c9-17d09ae1e05e\") " Jul 17 18:57:52.978720 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:52.978697 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a73792-4695-4e1d-a4c9-17d09ae1e05e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "64a73792-4695-4e1d-a4c9-17d09ae1e05e" (UID: "64a73792-4695-4e1d-a4c9-17d09ae1e05e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:57:53.079974 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:53.079939 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a73792-4695-4e1d-a4c9-17d09ae1e05e-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:57:53.737444 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:53.737413 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7d6f90-25tsp" Jul 17 18:57:53.737624 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:53.737406 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7d6f90-25tsp" event={"ID":"64a73792-4695-4e1d-a4c9-17d09ae1e05e","Type":"ContainerDied","Data":"ea72770efe430b9ce0bba0b48f0cf094364562a449e062e1f1327a3164ec840c"} Jul 17 18:57:53.737624 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:57:53.737534 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea72770efe430b9ce0bba0b48f0cf094364562a449e062e1f1327a3164ec840c" Jul 17 18:58:01.711919 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:01.711881 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw"] Jul 17 18:58:01.712340 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:01.712216 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64a73792-4695-4e1d-a4c9-17d09ae1e05e" containerName="revoke-keys" Jul 17 18:58:01.712340 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:01.712228 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="64a73792-4695-4e1d-a4c9-17d09ae1e05e" containerName="revoke-keys" Jul 17 18:58:01.712340 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:01.712238 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046a9907-a42b-4331-9aae-84c7aa4f2dad" containerName="revoke-keys" Jul 17 18:58:01.712340 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:01.712244 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="046a9907-a42b-4331-9aae-84c7aa4f2dad" containerName="revoke-keys" Jul 17 18:58:01.712340 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:01.712297 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="64a73792-4695-4e1d-a4c9-17d09ae1e05e" containerName="revoke-keys" Jul 17 18:58:01.715270 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:01.715254 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw" Jul 17 18:58:01.741501 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:01.741466 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw"] Jul 17 18:58:01.754756 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:01.754731 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55a9827-5d50-489c-98ee-58e01df2fbb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw\" (UID: \"e55a9827-5d50-489c-98ee-58e01df2fbb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw" Jul 17 18:58:01.856194 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:01.856158 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55a9827-5d50-489c-98ee-58e01df2fbb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw\" (UID: \"e55a9827-5d50-489c-98ee-58e01df2fbb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw" Jul 17 18:58:01.856771 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:01.856751 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55a9827-5d50-489c-98ee-58e01df2fbb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw\" (UID: \"e55a9827-5d50-489c-98ee-58e01df2fbb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw" Jul 17 18:58:02.024654 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:02.024565 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw" Jul 17 18:58:02.219844 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:02.219811 2572 scope.go:117] "RemoveContainer" containerID="47e008aba1d897f8f90ae337f616787b2ee1f549d86c7b52310407adb5c99df9" Jul 17 18:58:02.378957 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:02.378922 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw"] Jul 17 18:58:02.381933 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:58:02.381909 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55a9827_5d50_489c_98ee_58e01df2fbb8.slice/crio-9c3bdea03d82e7ed1921022ac428697fa1184311ed741bd491bd81d630aeb95a WatchSource:0}: Error finding container 9c3bdea03d82e7ed1921022ac428697fa1184311ed741bd491bd81d630aeb95a: Status 404 returned error can't find the container with id 9c3bdea03d82e7ed1921022ac428697fa1184311ed741bd491bd81d630aeb95a Jul 17 18:58:02.777701 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:02.777661 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k" event={"ID":"b1e5601d-4d66-44cd-a8df-f38885281dff","Type":"ContainerStarted","Data":"f54b40435ed97646b44508ba6356ad1a30489dccc79c9c8f932de6e88504aba9"} Jul 17 18:58:02.779165 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:02.779145 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw" event={"ID":"e55a9827-5d50-489c-98ee-58e01df2fbb8","Type":"ContainerStarted","Data":"7e705dd78488fe3f123f880f713278d686e748ada1147a7e12fd1f9dd699bb99"} Jul 17 18:58:02.779269 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:02.779186 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw" event={"ID":"e55a9827-5d50-489c-98ee-58e01df2fbb8","Type":"ContainerStarted","Data":"9c3bdea03d82e7ed1921022ac428697fa1184311ed741bd491bd81d630aeb95a"} Jul 17 18:58:02.814866 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:02.814818 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k" podStartSLOduration=16.814803191 podStartE2EDuration="16.814803191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:58:02.813628812 +0000 UTC m=+3678.223056419" watchObservedRunningTime="2026-07-17 18:58:02.814803191 +0000 UTC m=+3678.224230797" Jul 17 18:58:02.845669 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:02.845625 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw" podStartSLOduration=1.8456096770000001 podStartE2EDuration="1.845609677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:58:02.844797088 +0000 UTC m=+3678.254224693" watchObservedRunningTime="2026-07-17 18:58:02.845609677 +0000 UTC m=+3678.255037282" Jul 17 18:58:03.785892 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:03.785860 2572 generic.go:356] "Generic (PLEG): container finished" podID="b1e5601d-4d66-44cd-a8df-f38885281dff" containerID="f54b40435ed97646b44508ba6356ad1a30489dccc79c9c8f932de6e88504aba9" exitCode=6 Jul 17 18:58:03.786397 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:03.785929 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k" event={"ID":"b1e5601d-4d66-44cd-a8df-f38885281dff","Type":"ContainerDied","Data":"f54b40435ed97646b44508ba6356ad1a30489dccc79c9c8f932de6e88504aba9"} Jul 17 18:58:03.786397 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:03.785986 2572 scope.go:117] "RemoveContainer" containerID="47e008aba1d897f8f90ae337f616787b2ee1f549d86c7b52310407adb5c99df9" Jul 17 18:58:03.786397 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:03.786313 2572 scope.go:117] "RemoveContainer" containerID="f54b40435ed97646b44508ba6356ad1a30489dccc79c9c8f932de6e88504aba9" Jul 17 18:58:03.786594 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:58:03.786574 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k_odh-ai-gateway-infra(b1e5601d-4d66-44cd-a8df-f38885281dff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k" podUID="b1e5601d-4d66-44cd-a8df-f38885281dff" Jul 17 18:58:03.787694 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:03.787673 2572 generic.go:356] "Generic (PLEG): container finished" podID="e55a9827-5d50-489c-98ee-58e01df2fbb8" containerID="7e705dd78488fe3f123f880f713278d686e748ada1147a7e12fd1f9dd699bb99" exitCode=6 Jul 17 18:58:03.787787 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:03.787757 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw" event={"ID":"e55a9827-5d50-489c-98ee-58e01df2fbb8","Type":"ContainerDied","Data":"7e705dd78488fe3f123f880f713278d686e748ada1147a7e12fd1f9dd699bb99"} Jul 17 18:58:03.787915 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:03.787901 2572 scope.go:117] "RemoveContainer" containerID="7e705dd78488fe3f123f880f713278d686e748ada1147a7e12fd1f9dd699bb99" Jul 17 18:58:03.839327 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:03.839306 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k"] Jul 17 18:58:04.252505 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:04.252422 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx"] Jul 17 18:58:04.257319 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:04.257297 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx" Jul 17 18:58:04.278498 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:04.275353 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx"] Jul 17 18:58:04.278498 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:04.276059 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60fb851-25b4-4f38-9b2d-c66237e6228f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx\" (UID: \"f60fb851-25b4-4f38-9b2d-c66237e6228f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx" Jul 17 18:58:04.377384 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:04.377353 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60fb851-25b4-4f38-9b2d-c66237e6228f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx\" (UID: \"f60fb851-25b4-4f38-9b2d-c66237e6228f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx" Jul 17 18:58:04.377921 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:04.377899 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60fb851-25b4-4f38-9b2d-c66237e6228f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx\" (UID: \"f60fb851-25b4-4f38-9b2d-c66237e6228f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx" Jul 17 18:58:04.567283 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:04.567256 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx" Jul 17 18:58:04.734682 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:58:04.734653 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60fb851_25b4_4f38_9b2d_c66237e6228f.slice/crio-613db6fd4f2dd24619e467eabded64f9286cd952c8617f72300806b27ca99cb3 WatchSource:0}: Error finding container 613db6fd4f2dd24619e467eabded64f9286cd952c8617f72300806b27ca99cb3: Status 404 returned error can't find the container with id 613db6fd4f2dd24619e467eabded64f9286cd952c8617f72300806b27ca99cb3 Jul 17 18:58:04.736671 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:04.736649 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx"] Jul 17 18:58:04.794294 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:04.794268 2572 generic.go:356] "Generic (PLEG): container finished" podID="e55a9827-5d50-489c-98ee-58e01df2fbb8" containerID="85c730483c04d8af87613ffde16c4c3c3bcbb1180222bb44ca28be1cb572ef05" exitCode=6 Jul 17 18:58:04.794661 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:04.794365 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw" event={"ID":"e55a9827-5d50-489c-98ee-58e01df2fbb8","Type":"ContainerDied","Data":"85c730483c04d8af87613ffde16c4c3c3bcbb1180222bb44ca28be1cb572ef05"} Jul 17 18:58:04.794661 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:04.794437 2572 scope.go:117] "RemoveContainer" containerID="7e705dd78488fe3f123f880f713278d686e748ada1147a7e12fd1f9dd699bb99" Jul 17 18:58:04.794661 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:04.794515 2572 scope.go:117] "RemoveContainer" containerID="85c730483c04d8af87613ffde16c4c3c3bcbb1180222bb44ca28be1cb572ef05" Jul 17 18:58:04.794830 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:58:04.794710 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw_odh-ai-gateway-infra(e55a9827-5d50-489c-98ee-58e01df2fbb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw" podUID="e55a9827-5d50-489c-98ee-58e01df2fbb8" Jul 17 18:58:04.795619 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:04.795600 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx" event={"ID":"f60fb851-25b4-4f38-9b2d-c66237e6228f","Type":"ContainerStarted","Data":"613db6fd4f2dd24619e467eabded64f9286cd952c8617f72300806b27ca99cb3"} Jul 17 18:58:04.911598 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:04.911572 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k" Jul 17 18:58:04.981786 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:04.981757 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e5601d-4d66-44cd-a8df-f38885281dff-maas-api-service-ca\") pod \"b1e5601d-4d66-44cd-a8df-f38885281dff\" (UID: \"b1e5601d-4d66-44cd-a8df-f38885281dff\") " Jul 17 18:58:04.982157 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:04.982135 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1e5601d-4d66-44cd-a8df-f38885281dff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b1e5601d-4d66-44cd-a8df-f38885281dff" (UID: "b1e5601d-4d66-44cd-a8df-f38885281dff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:58:05.082993 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:05.082966 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1e5601d-4d66-44cd-a8df-f38885281dff-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:58:05.804151 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:05.804067 2572 scope.go:117] "RemoveContainer" containerID="85c730483c04d8af87613ffde16c4c3c3bcbb1180222bb44ca28be1cb572ef05" Jul 17 18:58:05.804572 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:58:05.804295 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw_odh-ai-gateway-infra(e55a9827-5d50-489c-98ee-58e01df2fbb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw" podUID="e55a9827-5d50-489c-98ee-58e01df2fbb8" Jul 17 18:58:05.805405 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:05.805384 2572 generic.go:356] "Generic (PLEG): container finished" podID="f60fb851-25b4-4f38-9b2d-c66237e6228f" containerID="0061acd4191ac218d3de85ddbfc589d4c6f4bad41bdf2d17d040acae47b86ffb" exitCode=6 Jul 17 18:58:05.805498 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:05.805468 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx" event={"ID":"f60fb851-25b4-4f38-9b2d-c66237e6228f","Type":"ContainerDied","Data":"0061acd4191ac218d3de85ddbfc589d4c6f4bad41bdf2d17d040acae47b86ffb"} Jul 17 18:58:05.805598 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:05.805584 2572 scope.go:117] "RemoveContainer" containerID="0061acd4191ac218d3de85ddbfc589d4c6f4bad41bdf2d17d040acae47b86ffb" Jul 17 18:58:05.806667 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:05.806647 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k" event={"ID":"b1e5601d-4d66-44cd-a8df-f38885281dff","Type":"ContainerDied","Data":"ca0c6ce26c32e768dc012b9cc7c2bd685a7aeabd93ea6b9637b90592d4529d28"} Jul 17 18:58:05.806667 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:05.806662 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k" Jul 17 18:58:05.806830 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:05.806694 2572 scope.go:117] "RemoveContainer" containerID="f54b40435ed97646b44508ba6356ad1a30489dccc79c9c8f932de6e88504aba9" Jul 17 18:58:05.857512 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:05.857490 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k"] Jul 17 18:58:05.866202 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:05.866181 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-c4h4k"] Jul 17 18:58:06.813548 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:06.813509 2572 generic.go:356] "Generic (PLEG): container finished" podID="f60fb851-25b4-4f38-9b2d-c66237e6228f" containerID="2f36506c07303e4691dc7d623b5f9d3e5a324d6622d5182a192fa0fc9807a8ac" exitCode=6 Jul 17 18:58:06.814175 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:06.813599 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx" event={"ID":"f60fb851-25b4-4f38-9b2d-c66237e6228f","Type":"ContainerDied","Data":"2f36506c07303e4691dc7d623b5f9d3e5a324d6622d5182a192fa0fc9807a8ac"} Jul 17 18:58:06.814175 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:06.813659 2572 scope.go:117] "RemoveContainer" containerID="0061acd4191ac218d3de85ddbfc589d4c6f4bad41bdf2d17d040acae47b86ffb" Jul 17 18:58:06.814175 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:06.813888 2572 scope.go:117] "RemoveContainer" containerID="2f36506c07303e4691dc7d623b5f9d3e5a324d6622d5182a192fa0fc9807a8ac" Jul 17 18:58:06.814175 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:58:06.814153 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx_odh-ai-gateway-infra(f60fb851-25b4-4f38-9b2d-c66237e6228f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx" podUID="f60fb851-25b4-4f38-9b2d-c66237e6228f" Jul 17 18:58:07.224761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:07.224685 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1e5601d-4d66-44cd-a8df-f38885281dff" path="/var/lib/kubelet/pods/b1e5601d-4d66-44cd-a8df-f38885281dff/volumes" Jul 17 18:58:07.821659 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:07.821629 2572 scope.go:117] "RemoveContainer" containerID="2f36506c07303e4691dc7d623b5f9d3e5a324d6622d5182a192fa0fc9807a8ac" Jul 17 18:58:07.822118 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:58:07.821866 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx_odh-ai-gateway-infra(f60fb851-25b4-4f38-9b2d-c66237e6228f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx" podUID="f60fb851-25b4-4f38-9b2d-c66237e6228f" Jul 17 18:58:16.554820 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:16.554776 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-52fc47-fmnbx"] Jul 17 18:58:16.555307 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:16.555233 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e5601d-4d66-44cd-a8df-f38885281dff" containerName="revoke-keys" Jul 17 18:58:16.555307 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:16.555250 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e5601d-4d66-44cd-a8df-f38885281dff" containerName="revoke-keys" Jul 17 18:58:16.555307 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:16.555262 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e5601d-4d66-44cd-a8df-f38885281dff" containerName="revoke-keys" Jul 17 18:58:16.555307 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:16.555270 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e5601d-4d66-44cd-a8df-f38885281dff" containerName="revoke-keys" Jul 17 18:58:16.555525 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:16.555366 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e5601d-4d66-44cd-a8df-f38885281dff" containerName="revoke-keys" Jul 17 18:58:16.555525 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:16.555380 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e5601d-4d66-44cd-a8df-f38885281dff" containerName="revoke-keys" Jul 17 18:58:16.559723 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:16.559702 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-52fc47-fmnbx" Jul 17 18:58:16.583959 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:16.583932 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-52fc47-fmnbx"] Jul 17 18:58:16.681350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:16.681320 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca034d20-b142-4e32-8139-40acc7f4a281-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-52fc47-fmnbx\" (UID: \"ca034d20-b142-4e32-8139-40acc7f4a281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-52fc47-fmnbx" Jul 17 18:58:16.782148 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:16.782115 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca034d20-b142-4e32-8139-40acc7f4a281-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-52fc47-fmnbx\" (UID: \"ca034d20-b142-4e32-8139-40acc7f4a281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-52fc47-fmnbx" Jul 17 18:58:16.782722 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:16.782704 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca034d20-b142-4e32-8139-40acc7f4a281-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-52fc47-fmnbx\" (UID: \"ca034d20-b142-4e32-8139-40acc7f4a281\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-52fc47-fmnbx" Jul 17 18:58:16.869392 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:16.869323 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-52fc47-fmnbx" Jul 17 18:58:17.015848 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:17.015822 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-52fc47-fmnbx"] Jul 17 18:58:17.018248 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:58:17.018223 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca034d20_b142_4e32_8139_40acc7f4a281.slice/crio-25558d34b6f04295c30e1d54b997ed99ffc25f64e4b24065a32b4e1ff678c74c WatchSource:0}: Error finding container 25558d34b6f04295c30e1d54b997ed99ffc25f64e4b24065a32b4e1ff678c74c: Status 404 returned error can't find the container with id 25558d34b6f04295c30e1d54b997ed99ffc25f64e4b24065a32b4e1ff678c74c Jul 17 18:58:17.870068 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:17.870036 2572 generic.go:356] "Generic (PLEG): container finished" podID="ca034d20-b142-4e32-8139-40acc7f4a281" containerID="fa19536e334c11700f5acd70dab1f99c649e676fc9e9c580af05452b7f1230c9" exitCode=0 Jul 17 18:58:17.870450 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:17.870124 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-52fc47-fmnbx" event={"ID":"ca034d20-b142-4e32-8139-40acc7f4a281","Type":"ContainerDied","Data":"fa19536e334c11700f5acd70dab1f99c649e676fc9e9c580af05452b7f1230c9"} Jul 17 18:58:17.870450 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:17.870176 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-52fc47-fmnbx" event={"ID":"ca034d20-b142-4e32-8139-40acc7f4a281","Type":"ContainerStarted","Data":"25558d34b6f04295c30e1d54b997ed99ffc25f64e4b24065a32b4e1ff678c74c"} Jul 17 18:58:18.283389 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:18.283355 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc"] Jul 17 18:58:18.283683 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:18.283671 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e5601d-4d66-44cd-a8df-f38885281dff" containerName="revoke-keys" Jul 17 18:58:18.283726 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:18.283684 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e5601d-4d66-44cd-a8df-f38885281dff" containerName="revoke-keys" Jul 17 18:58:18.283761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:18.283737 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e5601d-4d66-44cd-a8df-f38885281dff" containerName="revoke-keys" Jul 17 18:58:18.286629 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:18.286614 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc" Jul 17 18:58:18.305821 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:18.305794 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc"] Jul 17 18:58:18.397417 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:18.397383 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a46702-3aaf-445f-9f60-6d5c90634194-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc\" (UID: \"f8a46702-3aaf-445f-9f60-6d5c90634194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc" Jul 17 18:58:18.498211 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:18.498178 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a46702-3aaf-445f-9f60-6d5c90634194-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc\" (UID: \"f8a46702-3aaf-445f-9f60-6d5c90634194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc" Jul 17 18:58:18.498827 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:18.498808 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a46702-3aaf-445f-9f60-6d5c90634194-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc\" (UID: \"f8a46702-3aaf-445f-9f60-6d5c90634194\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc" Jul 17 18:58:18.596540 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:18.596462 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc" Jul 17 18:58:18.743034 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:18.742997 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc"] Jul 17 18:58:18.745466 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:58:18.745425 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a46702_3aaf_445f_9f60_6d5c90634194.slice/crio-61adf03c057828dd2d850aec17d6a9e8d142b8c109610eb8fe0c95dad3146e69 WatchSource:0}: Error finding container 61adf03c057828dd2d850aec17d6a9e8d142b8c109610eb8fe0c95dad3146e69: Status 404 returned error can't find the container with id 61adf03c057828dd2d850aec17d6a9e8d142b8c109610eb8fe0c95dad3146e69 Jul 17 18:58:18.876106 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:18.876037 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc" event={"ID":"f8a46702-3aaf-445f-9f60-6d5c90634194","Type":"ContainerStarted","Data":"61adf03c057828dd2d850aec17d6a9e8d142b8c109610eb8fe0c95dad3146e69"} Jul 17 18:58:19.040505 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:19.040474 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-52fc47-fmnbx" Jul 17 18:58:19.103533 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:19.103494 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca034d20-b142-4e32-8139-40acc7f4a281-maas-api-service-ca\") pod \"ca034d20-b142-4e32-8139-40acc7f4a281\" (UID: \"ca034d20-b142-4e32-8139-40acc7f4a281\") " Jul 17 18:58:19.103872 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:19.103849 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca034d20-b142-4e32-8139-40acc7f4a281-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ca034d20-b142-4e32-8139-40acc7f4a281" (UID: "ca034d20-b142-4e32-8139-40acc7f4a281"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:58:19.204647 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:19.204574 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca034d20-b142-4e32-8139-40acc7f4a281-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:58:19.219869 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:19.219843 2572 scope.go:117] "RemoveContainer" containerID="85c730483c04d8af87613ffde16c4c3c3bcbb1180222bb44ca28be1cb572ef05" Jul 17 18:58:19.881763 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:19.881688 2572 generic.go:356] "Generic (PLEG): container finished" podID="f8a46702-3aaf-445f-9f60-6d5c90634194" containerID="40f1ac2cfd54d7cf8afefeab4e3cd30f30bf275f4744dd2b180ca4a369c4f61e" exitCode=6 Jul 17 18:58:19.882176 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:19.881766 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc" event={"ID":"f8a46702-3aaf-445f-9f60-6d5c90634194","Type":"ContainerDied","Data":"40f1ac2cfd54d7cf8afefeab4e3cd30f30bf275f4744dd2b180ca4a369c4f61e"} Jul 17 18:58:19.882176 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:19.881940 2572 scope.go:117] "RemoveContainer" containerID="40f1ac2cfd54d7cf8afefeab4e3cd30f30bf275f4744dd2b180ca4a369c4f61e" Jul 17 18:58:19.883716 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:19.883697 2572 generic.go:356] "Generic (PLEG): container finished" podID="e55a9827-5d50-489c-98ee-58e01df2fbb8" containerID="e7084541f09d5a5297e20163ce7ee6c99ddb3f6808c920e6677dba4803d11bab" exitCode=6 Jul 17 18:58:19.883794 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:19.883763 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw" event={"ID":"e55a9827-5d50-489c-98ee-58e01df2fbb8","Type":"ContainerDied","Data":"e7084541f09d5a5297e20163ce7ee6c99ddb3f6808c920e6677dba4803d11bab"} Jul 17 18:58:19.883852 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:19.883806 2572 scope.go:117] "RemoveContainer" containerID="85c730483c04d8af87613ffde16c4c3c3bcbb1180222bb44ca28be1cb572ef05" Jul 17 18:58:19.884129 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:19.884112 2572 scope.go:117] "RemoveContainer" containerID="e7084541f09d5a5297e20163ce7ee6c99ddb3f6808c920e6677dba4803d11bab" Jul 17 18:58:19.884329 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:58:19.884309 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw_odh-ai-gateway-infra(e55a9827-5d50-489c-98ee-58e01df2fbb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw" podUID="e55a9827-5d50-489c-98ee-58e01df2fbb8" Jul 17 18:58:19.885163 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:19.885141 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-52fc47-fmnbx" event={"ID":"ca034d20-b142-4e32-8139-40acc7f4a281","Type":"ContainerDied","Data":"25558d34b6f04295c30e1d54b997ed99ffc25f64e4b24065a32b4e1ff678c74c"} Jul 17 18:58:19.885281 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:19.885169 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-52fc47-fmnbx" Jul 17 18:58:19.885281 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:19.885183 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25558d34b6f04295c30e1d54b997ed99ffc25f64e4b24065a32b4e1ff678c74c" Jul 17 18:58:20.561908 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:58:20.561884 2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a46702_3aaf_445f_9f60_6d5c90634194.slice/crio-conmon-20819566e28e014765ad17df1a0f846d2909b26fa9d799d8d650d8d08e9d99ce.scope\": RecentStats: unable to find data in memory cache]" Jul 17 18:58:20.891272 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:20.891187 2572 generic.go:356] "Generic (PLEG): container finished" podID="f8a46702-3aaf-445f-9f60-6d5c90634194" containerID="20819566e28e014765ad17df1a0f846d2909b26fa9d799d8d650d8d08e9d99ce" exitCode=6 Jul 17 18:58:20.891682 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:20.891277 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc" event={"ID":"f8a46702-3aaf-445f-9f60-6d5c90634194","Type":"ContainerDied","Data":"20819566e28e014765ad17df1a0f846d2909b26fa9d799d8d650d8d08e9d99ce"} Jul 17 18:58:20.891682 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:20.891333 2572 scope.go:117] "RemoveContainer" containerID="40f1ac2cfd54d7cf8afefeab4e3cd30f30bf275f4744dd2b180ca4a369c4f61e" Jul 17 18:58:20.891682 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:20.891512 2572 scope.go:117] "RemoveContainer" containerID="20819566e28e014765ad17df1a0f846d2909b26fa9d799d8d650d8d08e9d99ce" Jul 17 18:58:20.891812 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:58:20.891756 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc_odh-ai-gateway-infra(f8a46702-3aaf-445f-9f60-6d5c90634194)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc" podUID="f8a46702-3aaf-445f-9f60-6d5c90634194" Jul 17 18:58:20.976989 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:20.976952 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw"] Jul 17 18:58:21.111337 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:21.111311 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw" Jul 17 18:58:21.220259 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:21.220176 2572 scope.go:117] "RemoveContainer" containerID="2f36506c07303e4691dc7d623b5f9d3e5a324d6622d5182a192fa0fc9807a8ac" Jul 17 18:58:21.221151 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:21.221131 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55a9827-5d50-489c-98ee-58e01df2fbb8-maas-api-service-ca\") pod \"e55a9827-5d50-489c-98ee-58e01df2fbb8\" (UID: \"e55a9827-5d50-489c-98ee-58e01df2fbb8\") " Jul 17 18:58:21.221525 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:21.221498 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e55a9827-5d50-489c-98ee-58e01df2fbb8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e55a9827-5d50-489c-98ee-58e01df2fbb8" (UID: "e55a9827-5d50-489c-98ee-58e01df2fbb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:58:21.321922 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:21.321884 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55a9827-5d50-489c-98ee-58e01df2fbb8-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:58:21.899365 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:21.899277 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx" event={"ID":"f60fb851-25b4-4f38-9b2d-c66237e6228f","Type":"ContainerStarted","Data":"80807f7544a9ef3bff5a80445e6a8a377570b041b96b96d7df0e1b932ac52981"} Jul 17 18:58:21.900966 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:21.900944 2572 scope.go:117] "RemoveContainer" containerID="20819566e28e014765ad17df1a0f846d2909b26fa9d799d8d650d8d08e9d99ce" Jul 17 18:58:21.901159 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:58:21.901138 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc_odh-ai-gateway-infra(f8a46702-3aaf-445f-9f60-6d5c90634194)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc" podUID="f8a46702-3aaf-445f-9f60-6d5c90634194" Jul 17 18:58:21.901894 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:21.901880 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw" Jul 17 18:58:21.901894 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:21.901883 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw" event={"ID":"e55a9827-5d50-489c-98ee-58e01df2fbb8","Type":"ContainerDied","Data":"9c3bdea03d82e7ed1921022ac428697fa1184311ed741bd491bd81d630aeb95a"} Jul 17 18:58:21.902054 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:21.901928 2572 scope.go:117] "RemoveContainer" containerID="e7084541f09d5a5297e20163ce7ee6c99ddb3f6808c920e6677dba4803d11bab" Jul 17 18:58:21.945392 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:21.945347 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx" podStartSLOduration=17.945327903 podStartE2EDuration="17.945327903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:58:21.936051855 +0000 UTC m=+3697.345479482" watchObservedRunningTime="2026-07-17 18:58:21.945327903 +0000 UTC m=+3697.354755508" Jul 17 18:58:22.038083 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:22.038054 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw"] Jul 17 18:58:22.043196 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:22.043165 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-df78e9c5-b4tjw"] Jul 17 18:58:22.909042 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:22.908995 2572 generic.go:356] "Generic (PLEG): container finished" podID="f60fb851-25b4-4f38-9b2d-c66237e6228f" containerID="80807f7544a9ef3bff5a80445e6a8a377570b041b96b96d7df0e1b932ac52981" exitCode=6 Jul 17 18:58:22.909042 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:22.909037 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx" event={"ID":"f60fb851-25b4-4f38-9b2d-c66237e6228f","Type":"ContainerDied","Data":"80807f7544a9ef3bff5a80445e6a8a377570b041b96b96d7df0e1b932ac52981"} Jul 17 18:58:22.909530 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:22.909095 2572 scope.go:117] "RemoveContainer" containerID="2f36506c07303e4691dc7d623b5f9d3e5a324d6622d5182a192fa0fc9807a8ac" Jul 17 18:58:22.909530 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:22.909359 2572 scope.go:117] "RemoveContainer" containerID="80807f7544a9ef3bff5a80445e6a8a377570b041b96b96d7df0e1b932ac52981" Jul 17 18:58:22.909636 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:58:22.909597 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx_odh-ai-gateway-infra(f60fb851-25b4-4f38-9b2d-c66237e6228f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx" podUID="f60fb851-25b4-4f38-9b2d-c66237e6228f" Jul 17 18:58:22.970407 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:22.970375 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx"] Jul 17 18:58:23.224948 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:23.224871 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e55a9827-5d50-489c-98ee-58e01df2fbb8" path="/var/lib/kubelet/pods/e55a9827-5d50-489c-98ee-58e01df2fbb8/volumes" Jul 17 18:58:24.058137 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:24.058111 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx" Jul 17 18:58:24.147521 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:24.147416 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60fb851-25b4-4f38-9b2d-c66237e6228f-maas-api-service-ca\") pod \"f60fb851-25b4-4f38-9b2d-c66237e6228f\" (UID: \"f60fb851-25b4-4f38-9b2d-c66237e6228f\") " Jul 17 18:58:24.147737 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:24.147700 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f60fb851-25b4-4f38-9b2d-c66237e6228f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f60fb851-25b4-4f38-9b2d-c66237e6228f" (UID: "f60fb851-25b4-4f38-9b2d-c66237e6228f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:58:24.147855 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:24.147838 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60fb851-25b4-4f38-9b2d-c66237e6228f-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:58:24.922483 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:24.922452 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx" Jul 17 18:58:24.922483 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:24.922480 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx" event={"ID":"f60fb851-25b4-4f38-9b2d-c66237e6228f","Type":"ContainerDied","Data":"613db6fd4f2dd24619e467eabded64f9286cd952c8617f72300806b27ca99cb3"} Jul 17 18:58:24.922752 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:24.922534 2572 scope.go:117] "RemoveContainer" containerID="80807f7544a9ef3bff5a80445e6a8a377570b041b96b96d7df0e1b932ac52981" Jul 17 18:58:24.948615 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:24.948591 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx"] Jul 17 18:58:24.952245 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:24.952223 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-rc7lx"] Jul 17 18:58:25.224829 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:25.224752 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f60fb851-25b4-4f38-9b2d-c66237e6228f" path="/var/lib/kubelet/pods/f60fb851-25b4-4f38-9b2d-c66237e6228f/volumes" Jul 17 18:58:35.222059 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:35.222024 2572 scope.go:117] "RemoveContainer" containerID="20819566e28e014765ad17df1a0f846d2909b26fa9d799d8d650d8d08e9d99ce" Jul 17 18:58:35.974934 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:35.974900 2572 generic.go:356] "Generic (PLEG): container finished" podID="f8a46702-3aaf-445f-9f60-6d5c90634194" containerID="ab0072582028bc71ff3fec21dd2d305e17b70d169bc0a3f0d185eae6a3828983" exitCode=6 Jul 17 18:58:35.975101 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:35.974975 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc" event={"ID":"f8a46702-3aaf-445f-9f60-6d5c90634194","Type":"ContainerDied","Data":"ab0072582028bc71ff3fec21dd2d305e17b70d169bc0a3f0d185eae6a3828983"} Jul 17 18:58:35.975101 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:35.975051 2572 scope.go:117] "RemoveContainer" containerID="20819566e28e014765ad17df1a0f846d2909b26fa9d799d8d650d8d08e9d99ce" Jul 17 18:58:35.975348 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:35.975333 2572 scope.go:117] "RemoveContainer" containerID="ab0072582028bc71ff3fec21dd2d305e17b70d169bc0a3f0d185eae6a3828983" Jul 17 18:58:35.975608 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:58:35.975583 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc_odh-ai-gateway-infra(f8a46702-3aaf-445f-9f60-6d5c90634194)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc" podUID="f8a46702-3aaf-445f-9f60-6d5c90634194" Jul 17 18:58:36.181433 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181348 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj"] Jul 17 18:58:36.181668 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181655 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f60fb851-25b4-4f38-9b2d-c66237e6228f" containerName="revoke-keys" Jul 17 18:58:36.181668 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181668 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f60fb851-25b4-4f38-9b2d-c66237e6228f" containerName="revoke-keys" Jul 17 18:58:36.181761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181681 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e55a9827-5d50-489c-98ee-58e01df2fbb8" containerName="revoke-keys" Jul 17 18:58:36.181761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181687 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e55a9827-5d50-489c-98ee-58e01df2fbb8" containerName="revoke-keys" Jul 17 18:58:36.181761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181697 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f60fb851-25b4-4f38-9b2d-c66237e6228f" containerName="revoke-keys" Jul 17 18:58:36.181761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181702 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f60fb851-25b4-4f38-9b2d-c66237e6228f" containerName="revoke-keys" Jul 17 18:58:36.181761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181711 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e55a9827-5d50-489c-98ee-58e01df2fbb8" containerName="revoke-keys" Jul 17 18:58:36.181761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181716 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e55a9827-5d50-489c-98ee-58e01df2fbb8" containerName="revoke-keys" Jul 17 18:58:36.181761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181722 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e55a9827-5d50-489c-98ee-58e01df2fbb8" containerName="revoke-keys" Jul 17 18:58:36.181761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181726 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e55a9827-5d50-489c-98ee-58e01df2fbb8" containerName="revoke-keys" Jul 17 18:58:36.181761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181733 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f60fb851-25b4-4f38-9b2d-c66237e6228f" containerName="revoke-keys" Jul 17 18:58:36.181761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181738 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f60fb851-25b4-4f38-9b2d-c66237e6228f" containerName="revoke-keys" Jul 17 18:58:36.181761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181749 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca034d20-b142-4e32-8139-40acc7f4a281" containerName="revoke-keys" Jul 17 18:58:36.181761 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181754 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca034d20-b142-4e32-8139-40acc7f4a281" containerName="revoke-keys" Jul 17 18:58:36.182172 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181800 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e55a9827-5d50-489c-98ee-58e01df2fbb8" containerName="revoke-keys" Jul 17 18:58:36.182172 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181807 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca034d20-b142-4e32-8139-40acc7f4a281" containerName="revoke-keys" Jul 17 18:58:36.182172 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181813 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f60fb851-25b4-4f38-9b2d-c66237e6228f" containerName="revoke-keys" Jul 17 18:58:36.182172 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181819 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f60fb851-25b4-4f38-9b2d-c66237e6228f" containerName="revoke-keys" Jul 17 18:58:36.182172 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181823 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f60fb851-25b4-4f38-9b2d-c66237e6228f" containerName="revoke-keys" Jul 17 18:58:36.182172 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.181830 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e55a9827-5d50-489c-98ee-58e01df2fbb8" containerName="revoke-keys" Jul 17 18:58:36.185871 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.185854 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj" Jul 17 18:58:36.212350 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.212323 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj"] Jul 17 18:58:36.352056 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.352003 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30c46ce-aa9c-4514-a5f8-caec3b062473-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj\" (UID: \"a30c46ce-aa9c-4514-a5f8-caec3b062473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj" Jul 17 18:58:36.453470 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.453393 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30c46ce-aa9c-4514-a5f8-caec3b062473-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj\" (UID: \"a30c46ce-aa9c-4514-a5f8-caec3b062473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj" Jul 17 18:58:36.453989 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.453970 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30c46ce-aa9c-4514-a5f8-caec3b062473-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj\" (UID: \"a30c46ce-aa9c-4514-a5f8-caec3b062473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj" Jul 17 18:58:36.495497 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.495471 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj" Jul 17 18:58:36.639283 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.639257 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj"] Jul 17 18:58:36.641547 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:58:36.641517 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30c46ce_aa9c_4514_a5f8_caec3b062473.slice/crio-650c3a1b97e634a3f5bb8fb3c4e2ef0a66acb905639ac65fd3f18af5721acfc3 WatchSource:0}: Error finding container 650c3a1b97e634a3f5bb8fb3c4e2ef0a66acb905639ac65fd3f18af5721acfc3: Status 404 returned error can't find the container with id 650c3a1b97e634a3f5bb8fb3c4e2ef0a66acb905639ac65fd3f18af5721acfc3 Jul 17 18:58:36.982837 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.982803 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj" event={"ID":"a30c46ce-aa9c-4514-a5f8-caec3b062473","Type":"ContainerStarted","Data":"74a03f11c11054dc2b16dcc1fe39f75b28fb223805a9e415970195e73ac7f5b8"} Jul 17 18:58:36.983004 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:36.982854 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj" event={"ID":"a30c46ce-aa9c-4514-a5f8-caec3b062473","Type":"ContainerStarted","Data":"650c3a1b97e634a3f5bb8fb3c4e2ef0a66acb905639ac65fd3f18af5721acfc3"} Jul 17 18:58:37.055803 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:37.055771 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc"] Jul 17 18:58:37.067373 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:37.067321 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj" podStartSLOduration=1.067302901 podStartE2EDuration="1.067302901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:58:37.06724963 +0000 UTC m=+3712.476677237" watchObservedRunningTime="2026-07-17 18:58:37.067302901 +0000 UTC m=+3712.476730507" Jul 17 18:58:37.170883 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:37.170861 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc" Jul 17 18:58:37.261192 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:37.261164 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a46702-3aaf-445f-9f60-6d5c90634194-maas-api-service-ca\") pod \"f8a46702-3aaf-445f-9f60-6d5c90634194\" (UID: \"f8a46702-3aaf-445f-9f60-6d5c90634194\") " Jul 17 18:58:37.261538 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:37.261517 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a46702-3aaf-445f-9f60-6d5c90634194-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8a46702-3aaf-445f-9f60-6d5c90634194" (UID: "f8a46702-3aaf-445f-9f60-6d5c90634194"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:58:37.362720 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:37.362650 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a46702-3aaf-445f-9f60-6d5c90634194-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:58:37.988670 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:37.988645 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc" Jul 17 18:58:37.988947 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:37.988642 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc" event={"ID":"f8a46702-3aaf-445f-9f60-6d5c90634194","Type":"ContainerDied","Data":"61adf03c057828dd2d850aec17d6a9e8d142b8c109610eb8fe0c95dad3146e69"} Jul 17 18:58:37.988947 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:37.988774 2572 scope.go:117] "RemoveContainer" containerID="ab0072582028bc71ff3fec21dd2d305e17b70d169bc0a3f0d185eae6a3828983" Jul 17 18:58:37.990270 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:37.990250 2572 generic.go:356] "Generic (PLEG): container finished" podID="a30c46ce-aa9c-4514-a5f8-caec3b062473" containerID="74a03f11c11054dc2b16dcc1fe39f75b28fb223805a9e415970195e73ac7f5b8" exitCode=6 Jul 17 18:58:37.990374 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:37.990341 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj" event={"ID":"a30c46ce-aa9c-4514-a5f8-caec3b062473","Type":"ContainerDied","Data":"74a03f11c11054dc2b16dcc1fe39f75b28fb223805a9e415970195e73ac7f5b8"} Jul 17 18:58:37.990566 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:37.990546 2572 scope.go:117] "RemoveContainer" containerID="74a03f11c11054dc2b16dcc1fe39f75b28fb223805a9e415970195e73ac7f5b8" Jul 17 18:58:38.051967 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:38.051929 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc"] Jul 17 18:58:38.061437 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:38.061415 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-kdzbc"] Jul 17 18:58:38.997746 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:38.997712 2572 generic.go:356] "Generic (PLEG): container finished" podID="a30c46ce-aa9c-4514-a5f8-caec3b062473" containerID="92d5931a7d2d094d09a522ebcc4406d9970cefeea641d9cf57b2bf42c037543d" exitCode=6 Jul 17 18:58:38.998179 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:38.997804 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj" event={"ID":"a30c46ce-aa9c-4514-a5f8-caec3b062473","Type":"ContainerDied","Data":"92d5931a7d2d094d09a522ebcc4406d9970cefeea641d9cf57b2bf42c037543d"} Jul 17 18:58:38.998179 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:38.997863 2572 scope.go:117] "RemoveContainer" containerID="74a03f11c11054dc2b16dcc1fe39f75b28fb223805a9e415970195e73ac7f5b8" Jul 17 18:58:38.998179 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:38.998074 2572 scope.go:117] "RemoveContainer" containerID="92d5931a7d2d094d09a522ebcc4406d9970cefeea641d9cf57b2bf42c037543d" Jul 17 18:58:38.998314 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:58:38.998285 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj_odh-ai-gateway-infra(a30c46ce-aa9c-4514-a5f8-caec3b062473)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj" podUID="a30c46ce-aa9c-4514-a5f8-caec3b062473" Jul 17 18:58:39.224177 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:39.224140 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8a46702-3aaf-445f-9f60-6d5c90634194" path="/var/lib/kubelet/pods/f8a46702-3aaf-445f-9f60-6d5c90634194/volumes" Jul 17 18:58:40.003461 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:40.003434 2572 scope.go:117] "RemoveContainer" containerID="92d5931a7d2d094d09a522ebcc4406d9970cefeea641d9cf57b2bf42c037543d" Jul 17 18:58:40.003867 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:58:40.003628 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj_odh-ai-gateway-infra(a30c46ce-aa9c-4514-a5f8-caec3b062473)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj" podUID="a30c46ce-aa9c-4514-a5f8-caec3b062473" Jul 17 18:58:50.822235 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:50.822203 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d8bdcf84-f82c8_378e4b36-e952-4175-80f4-5756ab57c91a/copy-manifests/0.log" Jul 17 18:58:50.846363 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:50.846332 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d8bdcf84-f82c8_378e4b36-e952-4175-80f4-5756ab57c91a/manager/0.log" Jul 17 18:58:51.220371 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:51.220292 2572 scope.go:117] "RemoveContainer" containerID="92d5931a7d2d094d09a522ebcc4406d9970cefeea641d9cf57b2bf42c037543d" Jul 17 18:58:51.456451 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:51.456411 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738550-9vbgr_915b4b0b-aa37-4cf6-855c-f52bf142a336/cleanup/0.log" Jul 17 18:58:51.578872 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:51.578842 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738565-rdx25_301834ae-965c-4282-b448-735e2b2aa5cb/cleanup/0.log" Jul 17 18:58:51.694906 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:51.694875 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-a03eb5-m675m_d73a8ea8-927b-4242-8bbf-2cfebcabea93/revoke-keys/0.log" Jul 17 18:58:51.811580 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:51.811546 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-9de544-xcxtw_66455439-835c-4a6b-bc36-c67b9df80148/revoke-keys/0.log" Jul 17 18:58:51.936577 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:51.936554 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-ddeacf-5x5vt_bdeee194-26c7-4e9e-bf37-7952198f083e/revoke-keys/0.log" Jul 17 18:58:52.063253 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:52.063220 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-3be95b-dzvdl_6648d623-a8f1-44bb-87d1-399f9bdf2d05/revoke-keys/0.log" Jul 17 18:58:52.063423 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:52.063310 2572 generic.go:356] "Generic (PLEG): container finished" podID="a30c46ce-aa9c-4514-a5f8-caec3b062473" containerID="3dc6a1d913afb7a42109e8ebc29d1ffcd5f74b9a98d1657ad5306453d909a223" exitCode=6 Jul 17 18:58:52.063423 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:52.063365 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj" event={"ID":"a30c46ce-aa9c-4514-a5f8-caec3b062473","Type":"ContainerDied","Data":"3dc6a1d913afb7a42109e8ebc29d1ffcd5f74b9a98d1657ad5306453d909a223"} Jul 17 18:58:52.063423 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:52.063412 2572 scope.go:117] "RemoveContainer" containerID="92d5931a7d2d094d09a522ebcc4406d9970cefeea641d9cf57b2bf42c037543d" Jul 17 18:58:52.063684 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:52.063659 2572 scope.go:117] "RemoveContainer" containerID="3dc6a1d913afb7a42109e8ebc29d1ffcd5f74b9a98d1657ad5306453d909a223" Jul 17 18:58:52.063898 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:58:52.063875 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj_odh-ai-gateway-infra(a30c46ce-aa9c-4514-a5f8-caec3b062473)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj" podUID="a30c46ce-aa9c-4514-a5f8-caec3b062473" Jul 17 18:58:52.181165 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:52.181087 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj_a30c46ce-aa9c-4514-a5f8-caec3b062473/revoke-keys/2.log" Jul 17 18:58:52.595973 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:52.595948 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-7d6f90-25tsp_64a73792-4695-4e1d-a4c9-17d09ae1e05e/revoke-keys/0.log" Jul 17 18:58:52.722124 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:52.722094 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-52fc47-fmnbx_ca034d20-b142-4e32-8139-40acc7f4a281/revoke-keys/0.log" Jul 17 18:58:53.112985 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:53.112949 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj"] Jul 17 18:58:53.238867 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:53.238843 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj" Jul 17 18:58:53.300099 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:53.300072 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30c46ce-aa9c-4514-a5f8-caec3b062473-maas-api-service-ca\") pod \"a30c46ce-aa9c-4514-a5f8-caec3b062473\" (UID: \"a30c46ce-aa9c-4514-a5f8-caec3b062473\") " Jul 17 18:58:53.300425 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:53.300403 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a30c46ce-aa9c-4514-a5f8-caec3b062473-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a30c46ce-aa9c-4514-a5f8-caec3b062473" (UID: "a30c46ce-aa9c-4514-a5f8-caec3b062473"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:58:53.401655 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:53.401579 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30c46ce-aa9c-4514-a5f8-caec3b062473-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:58:54.074797 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:54.074767 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj" Jul 17 18:58:54.074984 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:54.074796 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj" event={"ID":"a30c46ce-aa9c-4514-a5f8-caec3b062473","Type":"ContainerDied","Data":"650c3a1b97e634a3f5bb8fb3c4e2ef0a66acb905639ac65fd3f18af5721acfc3"} Jul 17 18:58:54.074984 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:54.074873 2572 scope.go:117] "RemoveContainer" containerID="3dc6a1d913afb7a42109e8ebc29d1ffcd5f74b9a98d1657ad5306453d909a223" Jul 17 18:58:54.110060 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:54.110036 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj"] Jul 17 18:58:54.117972 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:54.117951 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35dc36c4-8q5gj"] Jul 17 18:58:55.224814 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:55.224779 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a30c46ce-aa9c-4514-a5f8-caec3b062473" path="/var/lib/kubelet/pods/a30c46ce-aa9c-4514-a5f8-caec3b062473/volumes" Jul 17 18:58:55.885904 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:55.885869 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-7rf9z_aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76/discovery/0.log" Jul 17 18:58:56.160431 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:56.160353 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84bfff449b-t7wpv_23012aad-30af-44b0-af8e-f341abe9d845/kube-auth-proxy/0.log" Jul 17 18:58:57.580384 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:58:57.580356 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-vqhs8_f33ef96e-58b0-4e4a-b80c-48b1d55cb8b7/main/0.log" Jul 17 18:59:04.484402 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.484372 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk"] Jul 17 18:59:04.484781 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.484657 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a46702-3aaf-445f-9f60-6d5c90634194" containerName="revoke-keys" Jul 17 18:59:04.484781 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.484668 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a46702-3aaf-445f-9f60-6d5c90634194" containerName="revoke-keys" Jul 17 18:59:04.484781 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.484679 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a30c46ce-aa9c-4514-a5f8-caec3b062473" containerName="revoke-keys" Jul 17 18:59:04.484781 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.484684 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a30c46ce-aa9c-4514-a5f8-caec3b062473" containerName="revoke-keys" Jul 17 18:59:04.484781 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.484691 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a30c46ce-aa9c-4514-a5f8-caec3b062473" containerName="revoke-keys" Jul 17 18:59:04.484781 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.484696 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a30c46ce-aa9c-4514-a5f8-caec3b062473" containerName="revoke-keys" Jul 17 18:59:04.484781 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.484707 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a46702-3aaf-445f-9f60-6d5c90634194" containerName="revoke-keys" Jul 17 18:59:04.484781 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.484713 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a46702-3aaf-445f-9f60-6d5c90634194" containerName="revoke-keys" Jul 17 18:59:04.484781 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.484781 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e55a9827-5d50-489c-98ee-58e01df2fbb8" containerName="revoke-keys" Jul 17 18:59:04.485124 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.484789 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a46702-3aaf-445f-9f60-6d5c90634194" containerName="revoke-keys" Jul 17 18:59:04.485124 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.484796 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a30c46ce-aa9c-4514-a5f8-caec3b062473" containerName="revoke-keys" Jul 17 18:59:04.485124 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.484802 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a46702-3aaf-445f-9f60-6d5c90634194" containerName="revoke-keys" Jul 17 18:59:04.485124 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.484808 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a30c46ce-aa9c-4514-a5f8-caec3b062473" containerName="revoke-keys" Jul 17 18:59:04.487500 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.487483 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk" Jul 17 18:59:04.495167 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.495148 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8m6gj\"" Jul 17 18:59:04.495167 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.495165 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:59:04.597489 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.597449 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f545fce-cdfb-497f-99a3-fdf207afa91c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk\" (UID: \"0f545fce-cdfb-497f-99a3-fdf207afa91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk" Jul 17 18:59:04.650366 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.650335 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk"] Jul 17 18:59:04.698438 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.698413 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f545fce-cdfb-497f-99a3-fdf207afa91c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk\" (UID: \"0f545fce-cdfb-497f-99a3-fdf207afa91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk" Jul 17 18:59:04.699026 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.698994 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f545fce-cdfb-497f-99a3-fdf207afa91c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk\" (UID: \"0f545fce-cdfb-497f-99a3-fdf207afa91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk" Jul 17 18:59:04.797254 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.797224 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk" Jul 17 18:59:04.965746 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:59:04.965721 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f545fce_cdfb_497f_99a3_fdf207afa91c.slice/crio-bd41b1d9846185df0e9306a24df8b1ed061efb9a473a552dc5c7f90f52d6e8ac WatchSource:0}: Error finding container bd41b1d9846185df0e9306a24df8b1ed061efb9a473a552dc5c7f90f52d6e8ac: Status 404 returned error can't find the container with id bd41b1d9846185df0e9306a24df8b1ed061efb9a473a552dc5c7f90f52d6e8ac Jul 17 18:59:04.966907 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:04.966881 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk"] Jul 17 18:59:05.125956 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:05.125874 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk" event={"ID":"0f545fce-cdfb-497f-99a3-fdf207afa91c","Type":"ContainerStarted","Data":"bd41b1d9846185df0e9306a24df8b1ed061efb9a473a552dc5c7f90f52d6e8ac"} Jul 17 18:59:06.131724 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:06.131691 2572 generic.go:356] "Generic (PLEG): container finished" podID="0f545fce-cdfb-497f-99a3-fdf207afa91c" containerID="0d9113b892f63f5fd09ced136fe12fea8f59d4701490fc111f78c531f5ed8c18" exitCode=6 Jul 17 18:59:06.132148 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:06.131741 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk" event={"ID":"0f545fce-cdfb-497f-99a3-fdf207afa91c","Type":"ContainerDied","Data":"0d9113b892f63f5fd09ced136fe12fea8f59d4701490fc111f78c531f5ed8c18"} Jul 17 18:59:06.132148 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:06.131971 2572 scope.go:117] "RemoveContainer" containerID="0d9113b892f63f5fd09ced136fe12fea8f59d4701490fc111f78c531f5ed8c18" Jul 17 18:59:07.137804 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:07.137769 2572 generic.go:356] "Generic (PLEG): container finished" podID="0f545fce-cdfb-497f-99a3-fdf207afa91c" containerID="a079954d18d9e6ce02a74f00e54c1d4cc1cd1720227445206ca3bbfecf60504b" exitCode=6 Jul 17 18:59:07.138241 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:07.137852 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk" event={"ID":"0f545fce-cdfb-497f-99a3-fdf207afa91c","Type":"ContainerDied","Data":"a079954d18d9e6ce02a74f00e54c1d4cc1cd1720227445206ca3bbfecf60504b"} Jul 17 18:59:07.138241 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:07.137913 2572 scope.go:117] "RemoveContainer" containerID="0d9113b892f63f5fd09ced136fe12fea8f59d4701490fc111f78c531f5ed8c18" Jul 17 18:59:07.138241 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:07.138199 2572 scope.go:117] "RemoveContainer" containerID="a079954d18d9e6ce02a74f00e54c1d4cc1cd1720227445206ca3bbfecf60504b" Jul 17 18:59:07.138411 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:59:07.138394 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk_odh-ai-gateway-infra(0f545fce-cdfb-497f-99a3-fdf207afa91c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk" podUID="0f545fce-cdfb-497f-99a3-fdf207afa91c" Jul 17 18:59:08.144640 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:08.144611 2572 scope.go:117] "RemoveContainer" containerID="a079954d18d9e6ce02a74f00e54c1d4cc1cd1720227445206ca3bbfecf60504b" Jul 17 18:59:08.144994 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:59:08.144812 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk_odh-ai-gateway-infra(0f545fce-cdfb-497f-99a3-fdf207afa91c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk" podUID="0f545fce-cdfb-497f-99a3-fdf207afa91c" Jul 17 18:59:08.365130 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:08.365099 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ts454_118b7937-fc4d-48fa-927e-2368a2317850/global-pull-secret-syncer/0.log" Jul 17 18:59:08.384087 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:08.384055 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs"] Jul 17 18:59:08.384393 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:08.384381 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a46702-3aaf-445f-9f60-6d5c90634194" containerName="revoke-keys" Jul 17 18:59:08.384439 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:08.384394 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a46702-3aaf-445f-9f60-6d5c90634194" containerName="revoke-keys" Jul 17 18:59:08.384439 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:08.384403 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a30c46ce-aa9c-4514-a5f8-caec3b062473" containerName="revoke-keys" Jul 17 18:59:08.384439 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:08.384409 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a30c46ce-aa9c-4514-a5f8-caec3b062473" containerName="revoke-keys" Jul 17 18:59:08.384530 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:08.384466 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a30c46ce-aa9c-4514-a5f8-caec3b062473" containerName="revoke-keys" Jul 17 18:59:08.384530 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:08.384474 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a46702-3aaf-445f-9f60-6d5c90634194" containerName="revoke-keys" Jul 17 18:59:08.388309 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:08.388292 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs" Jul 17 18:59:08.412966 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:08.412886 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs"] Jul 17 18:59:08.533190 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:08.533156 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cb9f3a2-74bf-45f5-9945-75ffdd80a501-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs\" (UID: \"4cb9f3a2-74bf-45f5-9945-75ffdd80a501\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs" Jul 17 18:59:08.634171 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:08.634139 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cb9f3a2-74bf-45f5-9945-75ffdd80a501-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs\" (UID: \"4cb9f3a2-74bf-45f5-9945-75ffdd80a501\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs" Jul 17 18:59:08.634753 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:08.634728 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cb9f3a2-74bf-45f5-9945-75ffdd80a501-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs\" (UID: \"4cb9f3a2-74bf-45f5-9945-75ffdd80a501\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs" Jul 17 18:59:08.648564 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:08.648535 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xbw8z_48aa1cde-e776-496a-9816-9a67614f0210/konnectivity-agent/0.log" Jul 17 18:59:08.697883 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:08.697831 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs" Jul 17 18:59:08.751188 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:08.751165 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-62.ec2.internal_8b28b739fa6bf195253cfa6375ca90e6/haproxy/0.log" Jul 17 18:59:08.842448 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:08.842281 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs"] Jul 17 18:59:08.844951 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:59:08.844924 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb9f3a2_74bf_45f5_9945_75ffdd80a501.slice/crio-8a1eeb101ce9eff6ca7659226c1e4b264c51bc3f2f32ac67a13e337150a37e7e WatchSource:0}: Error finding container 8a1eeb101ce9eff6ca7659226c1e4b264c51bc3f2f32ac67a13e337150a37e7e: Status 404 returned error can't find the container with id 8a1eeb101ce9eff6ca7659226c1e4b264c51bc3f2f32ac67a13e337150a37e7e Jul 17 18:59:09.150181 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:09.150151 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs" event={"ID":"4cb9f3a2-74bf-45f5-9945-75ffdd80a501","Type":"ContainerStarted","Data":"4e5a698242bef89dbc82c6807e7c2e710e2a2e8cc72482d49c19cdad6acd03f0"} Jul 17 18:59:09.150604 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:09.150203 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs" event={"ID":"4cb9f3a2-74bf-45f5-9945-75ffdd80a501","Type":"ContainerStarted","Data":"8a1eeb101ce9eff6ca7659226c1e4b264c51bc3f2f32ac67a13e337150a37e7e"} Jul 17 18:59:09.197553 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:09.197501 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs" podStartSLOduration=1.197485329 podStartE2EDuration="1.197485329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:59:09.19615441 +0000 UTC m=+3744.605582016" watchObservedRunningTime="2026-07-17 18:59:09.197485329 +0000 UTC m=+3744.606913175" Jul 17 18:59:10.155999 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:10.155965 2572 generic.go:356] "Generic (PLEG): container finished" podID="4cb9f3a2-74bf-45f5-9945-75ffdd80a501" containerID="4e5a698242bef89dbc82c6807e7c2e710e2a2e8cc72482d49c19cdad6acd03f0" exitCode=6 Jul 17 18:59:10.156436 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:10.156053 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs" event={"ID":"4cb9f3a2-74bf-45f5-9945-75ffdd80a501","Type":"ContainerDied","Data":"4e5a698242bef89dbc82c6807e7c2e710e2a2e8cc72482d49c19cdad6acd03f0"} Jul 17 18:59:10.156436 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:10.156216 2572 scope.go:117] "RemoveContainer" containerID="4e5a698242bef89dbc82c6807e7c2e710e2a2e8cc72482d49c19cdad6acd03f0" Jul 17 18:59:11.162772 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:11.162737 2572 generic.go:356] "Generic (PLEG): container finished" podID="4cb9f3a2-74bf-45f5-9945-75ffdd80a501" containerID="52dccd4fbad679be31917ae233e22ac1bcd41d111af87ae57c8a3c6e4333025f" exitCode=6 Jul 17 18:59:11.163269 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:11.162825 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs" event={"ID":"4cb9f3a2-74bf-45f5-9945-75ffdd80a501","Type":"ContainerDied","Data":"52dccd4fbad679be31917ae233e22ac1bcd41d111af87ae57c8a3c6e4333025f"} Jul 17 18:59:11.163269 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:11.162899 2572 scope.go:117] "RemoveContainer" containerID="4e5a698242bef89dbc82c6807e7c2e710e2a2e8cc72482d49c19cdad6acd03f0" Jul 17 18:59:11.163269 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:11.163145 2572 scope.go:117] "RemoveContainer" containerID="52dccd4fbad679be31917ae233e22ac1bcd41d111af87ae57c8a3c6e4333025f" Jul 17 18:59:11.163412 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:59:11.163392 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs_odh-ai-gateway-infra(4cb9f3a2-74bf-45f5-9945-75ffdd80a501)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs" podUID="4cb9f3a2-74bf-45f5-9945-75ffdd80a501" Jul 17 18:59:12.169137 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:12.169111 2572 scope.go:117] "RemoveContainer" containerID="52dccd4fbad679be31917ae233e22ac1bcd41d111af87ae57c8a3c6e4333025f" Jul 17 18:59:12.169572 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:59:12.169283 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs_odh-ai-gateway-infra(4cb9f3a2-74bf-45f5-9945-75ffdd80a501)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs" podUID="4cb9f3a2-74bf-45f5-9945-75ffdd80a501" Jul 17 18:59:16.841133 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:16.841105 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-84cfcc445d-gxlq7_1a321161-6c33-48dd-8388-b39ed73c2379/metrics-server/0.log" Jul 17 18:59:17.330889 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.330862 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rfp68_1f293570-245b-432e-b029-31aa06cd7156/node-exporter/0.log" Jul 17 18:59:17.382306 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.382277 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rfp68_1f293570-245b-432e-b029-31aa06cd7156/kube-rbac-proxy/0.log" Jul 17 18:59:17.449297 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.449272 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rfp68_1f293570-245b-432e-b029-31aa06cd7156/init-textfile/0.log" Jul 17 18:59:17.607668 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.607586 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689"] Jul 17 18:59:17.612190 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.612174 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:17.619803 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.619781 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lz7w4\"/\"default-dockercfg-ft9qh\"" Jul 17 18:59:17.620696 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.620673 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lz7w4\"/\"openshift-service-ca.crt\"" Jul 17 18:59:17.620696 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.620691 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lz7w4\"/\"kube-root-ca.crt\"" Jul 17 18:59:17.647562 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.647532 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689"] Jul 17 18:59:17.716676 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.716647 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a105182-328b-45bf-8c4c-0c2efdf6a8c7-podres\") pod \"perf-node-gather-daemonset-4m689\" (UID: \"6a105182-328b-45bf-8c4c-0c2efdf6a8c7\") " pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:17.716882 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.716694 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a105182-328b-45bf-8c4c-0c2efdf6a8c7-sys\") pod \"perf-node-gather-daemonset-4m689\" (UID: \"6a105182-328b-45bf-8c4c-0c2efdf6a8c7\") " pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:17.716882 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.716712 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a105182-328b-45bf-8c4c-0c2efdf6a8c7-proc\") pod \"perf-node-gather-daemonset-4m689\" (UID: \"6a105182-328b-45bf-8c4c-0c2efdf6a8c7\") " pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:17.716882 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.716727 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps82l\" (UniqueName: \"kubernetes.io/projected/6a105182-328b-45bf-8c4c-0c2efdf6a8c7-kube-api-access-ps82l\") pod \"perf-node-gather-daemonset-4m689\" (UID: \"6a105182-328b-45bf-8c4c-0c2efdf6a8c7\") " pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:17.716882 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.716797 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a105182-328b-45bf-8c4c-0c2efdf6a8c7-lib-modules\") pod \"perf-node-gather-daemonset-4m689\" (UID: \"6a105182-328b-45bf-8c4c-0c2efdf6a8c7\") " pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:17.767061 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.767018 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb/prometheus/0.log" Jul 17 18:59:17.817687 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.817657 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a105182-328b-45bf-8c4c-0c2efdf6a8c7-sys\") pod \"perf-node-gather-daemonset-4m689\" (UID: \"6a105182-328b-45bf-8c4c-0c2efdf6a8c7\") " pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:17.817687 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.817687 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a105182-328b-45bf-8c4c-0c2efdf6a8c7-proc\") pod \"perf-node-gather-daemonset-4m689\" (UID: \"6a105182-328b-45bf-8c4c-0c2efdf6a8c7\") " pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:17.817886 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.817705 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps82l\" (UniqueName: \"kubernetes.io/projected/6a105182-328b-45bf-8c4c-0c2efdf6a8c7-kube-api-access-ps82l\") pod \"perf-node-gather-daemonset-4m689\" (UID: \"6a105182-328b-45bf-8c4c-0c2efdf6a8c7\") " pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:17.817886 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.817784 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a105182-328b-45bf-8c4c-0c2efdf6a8c7-sys\") pod \"perf-node-gather-daemonset-4m689\" (UID: \"6a105182-328b-45bf-8c4c-0c2efdf6a8c7\") " pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:17.817886 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.817785 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a105182-328b-45bf-8c4c-0c2efdf6a8c7-proc\") pod \"perf-node-gather-daemonset-4m689\" (UID: \"6a105182-328b-45bf-8c4c-0c2efdf6a8c7\") " pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:17.817886 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.817824 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a105182-328b-45bf-8c4c-0c2efdf6a8c7-lib-modules\") pod \"perf-node-gather-daemonset-4m689\" (UID: \"6a105182-328b-45bf-8c4c-0c2efdf6a8c7\") " pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:17.817886 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.817877 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a105182-328b-45bf-8c4c-0c2efdf6a8c7-podres\") pod \"perf-node-gather-daemonset-4m689\" (UID: \"6a105182-328b-45bf-8c4c-0c2efdf6a8c7\") " pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:17.818095 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.817980 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a105182-328b-45bf-8c4c-0c2efdf6a8c7-lib-modules\") pod \"perf-node-gather-daemonset-4m689\" (UID: \"6a105182-328b-45bf-8c4c-0c2efdf6a8c7\") " pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:17.818095 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.818030 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a105182-328b-45bf-8c4c-0c2efdf6a8c7-podres\") pod \"perf-node-gather-daemonset-4m689\" (UID: \"6a105182-328b-45bf-8c4c-0c2efdf6a8c7\") " pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:17.831790 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.831771 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps82l\" (UniqueName: \"kubernetes.io/projected/6a105182-328b-45bf-8c4c-0c2efdf6a8c7-kube-api-access-ps82l\") pod \"perf-node-gather-daemonset-4m689\" (UID: \"6a105182-328b-45bf-8c4c-0c2efdf6a8c7\") " pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:17.896219 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.896138 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb/config-reloader/0.log" Jul 17 18:59:17.922202 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:17.922178 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:18.021194 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:18.021164 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb/thanos-sidecar/0.log" Jul 17 18:59:18.134956 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:18.134929 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689"] Jul 17 18:59:18.135986 ip-10-0-132-62 kubenswrapper[2572]: W0717 18:59:18.135958 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6a105182_328b_45bf_8c4c_0c2efdf6a8c7.slice/crio-625ab4fb0a4919ca967902c1dc683b269a3886e159c29edfffc737aed7d7d68a WatchSource:0}: Error finding container 625ab4fb0a4919ca967902c1dc683b269a3886e159c29edfffc737aed7d7d68a: Status 404 returned error can't find the container with id 625ab4fb0a4919ca967902c1dc683b269a3886e159c29edfffc737aed7d7d68a Jul 17 18:59:18.136905 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:18.136889 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb/kube-rbac-proxy-web/0.log" Jul 17 18:59:18.200028 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:18.199991 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" event={"ID":"6a105182-328b-45bf-8c4c-0c2efdf6a8c7","Type":"ContainerStarted","Data":"20db868a3909f36a1e0735c48efe84b7c3bd381c69848e0a52a28099e1e75741"} Jul 17 18:59:18.200155 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:18.200049 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" event={"ID":"6a105182-328b-45bf-8c4c-0c2efdf6a8c7","Type":"ContainerStarted","Data":"625ab4fb0a4919ca967902c1dc683b269a3886e159c29edfffc737aed7d7d68a"} Jul 17 18:59:18.200155 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:18.200121 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:18.270159 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:18.270131 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb/kube-rbac-proxy/0.log" Jul 17 18:59:18.272731 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:18.272691 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" podStartSLOduration=1.2726766729999999 podStartE2EDuration="1.272676673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:59:18.271326292 +0000 UTC m=+3753.680753897" watchObservedRunningTime="2026-07-17 18:59:18.272676673 +0000 UTC m=+3753.682104280" Jul 17 18:59:18.354528 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:18.354501 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb/kube-rbac-proxy-thanos/0.log" Jul 17 18:59:18.421944 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:18.421882 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_32c3e3ef-c984-4fe4-9bb4-5a90fcede3fb/init-config-reloader/0.log" Jul 17 18:59:18.625593 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:18.625566 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79bc7d6f4f-pwjtp_e75a9a63-f13c-49d9-b156-4d45362e0cfe/telemeter-client/0.log" Jul 17 18:59:18.660664 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:18.660640 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79bc7d6f4f-pwjtp_e75a9a63-f13c-49d9-b156-4d45362e0cfe/reload/0.log" Jul 17 18:59:18.701085 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:18.700995 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79bc7d6f4f-pwjtp_e75a9a63-f13c-49d9-b156-4d45362e0cfe/kube-rbac-proxy/0.log" Jul 17 18:59:21.220401 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:21.220376 2572 scope.go:117] "RemoveContainer" containerID="a079954d18d9e6ce02a74f00e54c1d4cc1cd1720227445206ca3bbfecf60504b" Jul 17 18:59:21.445060 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:21.445036 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-5v6b2_7a399b6a-5930-4576-9ec9-a1c6ccfc3a8f/download-server/0.log" Jul 17 18:59:22.219404 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:22.219319 2572 generic.go:356] "Generic (PLEG): container finished" podID="0f545fce-cdfb-497f-99a3-fdf207afa91c" containerID="6fa2a3da116bf1a7fe007ecb280527653cfa9056abfe93fa7fec7cafbc57e7cb" exitCode=6 Jul 17 18:59:22.219404 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:22.219383 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk" event={"ID":"0f545fce-cdfb-497f-99a3-fdf207afa91c","Type":"ContainerDied","Data":"6fa2a3da116bf1a7fe007ecb280527653cfa9056abfe93fa7fec7cafbc57e7cb"} Jul 17 18:59:22.219618 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:22.219442 2572 scope.go:117] "RemoveContainer" containerID="a079954d18d9e6ce02a74f00e54c1d4cc1cd1720227445206ca3bbfecf60504b" Jul 17 18:59:22.219717 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:22.219702 2572 scope.go:117] "RemoveContainer" containerID="6fa2a3da116bf1a7fe007ecb280527653cfa9056abfe93fa7fec7cafbc57e7cb" Jul 17 18:59:22.219932 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:59:22.219911 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk_odh-ai-gateway-infra(0f545fce-cdfb-497f-99a3-fdf207afa91c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk" podUID="0f545fce-cdfb-497f-99a3-fdf207afa91c" Jul 17 18:59:23.219908 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:23.219877 2572 scope.go:117] "RemoveContainer" containerID="52dccd4fbad679be31917ae233e22ac1bcd41d111af87ae57c8a3c6e4333025f" Jul 17 18:59:23.259835 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:23.259802 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk"] Jul 17 18:59:23.397224 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:23.397197 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk" Jul 17 18:59:23.466318 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:23.466292 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nxcs2_9eb2fc23-38d4-4bac-874b-706356e5f1f7/dns/0.log" Jul 17 18:59:23.504657 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:23.504595 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nxcs2_9eb2fc23-38d4-4bac-874b-706356e5f1f7/kube-rbac-proxy/0.log" Jul 17 18:59:23.572873 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:23.572840 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f545fce-cdfb-497f-99a3-fdf207afa91c-maas-api-service-ca\") pod \"0f545fce-cdfb-497f-99a3-fdf207afa91c\" (UID: \"0f545fce-cdfb-497f-99a3-fdf207afa91c\") " Jul 17 18:59:23.573193 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:23.573172 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f545fce-cdfb-497f-99a3-fdf207afa91c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0f545fce-cdfb-497f-99a3-fdf207afa91c" (UID: "0f545fce-cdfb-497f-99a3-fdf207afa91c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:59:23.591673 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:23.591647 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gpnwp_376644cc-94a0-41d2-9cc0-cc410806ecf8/dns-node-resolver/0.log" Jul 17 18:59:23.673573 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:23.673537 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f545fce-cdfb-497f-99a3-fdf207afa91c-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:59:24.213262 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:24.213191 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lz7w4/perf-node-gather-daemonset-4m689" Jul 17 18:59:24.231905 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:24.231876 2572 generic.go:356] "Generic (PLEG): container finished" podID="4cb9f3a2-74bf-45f5-9945-75ffdd80a501" containerID="a8cbdbf40c0b1a25257459c8c02e08e98b69379db92c3bb2d9ce7df384226657" exitCode=6 Jul 17 18:59:24.231905 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:24.231900 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs" event={"ID":"4cb9f3a2-74bf-45f5-9945-75ffdd80a501","Type":"ContainerDied","Data":"a8cbdbf40c0b1a25257459c8c02e08e98b69379db92c3bb2d9ce7df384226657"} Jul 17 18:59:24.232449 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:24.231964 2572 scope.go:117] "RemoveContainer" containerID="52dccd4fbad679be31917ae233e22ac1bcd41d111af87ae57c8a3c6e4333025f" Jul 17 18:59:24.232449 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:24.232372 2572 scope.go:117] "RemoveContainer" containerID="a8cbdbf40c0b1a25257459c8c02e08e98b69379db92c3bb2d9ce7df384226657" Jul 17 18:59:24.232677 ip-10-0-132-62 kubenswrapper[2572]: E0717 18:59:24.232658 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs_odh-ai-gateway-infra(4cb9f3a2-74bf-45f5-9945-75ffdd80a501)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs" podUID="4cb9f3a2-74bf-45f5-9945-75ffdd80a501" Jul 17 18:59:24.233413 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:24.233405 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk" Jul 17 18:59:24.233511 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:24.233442 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk" event={"ID":"0f545fce-cdfb-497f-99a3-fdf207afa91c","Type":"ContainerDied","Data":"bd41b1d9846185df0e9306a24df8b1ed061efb9a473a552dc5c7f90f52d6e8ac"} Jul 17 18:59:24.239338 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:24.239316 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7b558bd496-vjjrw_88632bb0-62df-4983-970b-ac75701af480/registry/0.log" Jul 17 18:59:24.243688 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:24.243670 2572 scope.go:117] "RemoveContainer" containerID="6fa2a3da116bf1a7fe007ecb280527653cfa9056abfe93fa7fec7cafbc57e7cb" Jul 17 18:59:24.284364 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:24.284331 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk"] Jul 17 18:59:24.288198 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:24.288172 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a5b144da-wxpgk"] Jul 17 18:59:24.333662 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:24.333636 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hfrgj_c8841f5d-e09f-4e42-b700-0002b5a3a342/node-ca/0.log" Jul 17 18:59:25.224935 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:25.224902 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f545fce-cdfb-497f-99a3-fdf207afa91c" path="/var/lib/kubelet/pods/0f545fce-cdfb-497f-99a3-fdf207afa91c/volumes" Jul 17 18:59:25.278902 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:25.278868 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs"] Jul 17 18:59:25.407129 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:25.407103 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs" Jul 17 18:59:25.591084 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:25.591051 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cb9f3a2-74bf-45f5-9945-75ffdd80a501-maas-api-service-ca\") pod \"4cb9f3a2-74bf-45f5-9945-75ffdd80a501\" (UID: \"4cb9f3a2-74bf-45f5-9945-75ffdd80a501\") " Jul 17 18:59:25.591407 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:25.591386 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb9f3a2-74bf-45f5-9945-75ffdd80a501-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4cb9f3a2-74bf-45f5-9945-75ffdd80a501" (UID: "4cb9f3a2-74bf-45f5-9945-75ffdd80a501"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:59:25.685979 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:25.685944 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-7rf9z_aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76/discovery/0.log" Jul 17 18:59:25.691977 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:25.691943 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cb9f3a2-74bf-45f5-9945-75ffdd80a501-maas-api-service-ca\") on node \"ip-10-0-132-62.ec2.internal\" DevicePath \"\"" Jul 17 18:59:25.752732 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:25.752683 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84bfff449b-t7wpv_23012aad-30af-44b0-af8e-f341abe9d845/kube-auth-proxy/0.log" Jul 17 18:59:26.245507 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:26.245479 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs" Jul 17 18:59:26.245685 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:26.245480 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs" event={"ID":"4cb9f3a2-74bf-45f5-9945-75ffdd80a501","Type":"ContainerDied","Data":"8a1eeb101ce9eff6ca7659226c1e4b264c51bc3f2f32ac67a13e337150a37e7e"} Jul 17 18:59:26.245685 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:26.245613 2572 scope.go:117] "RemoveContainer" containerID="a8cbdbf40c0b1a25257459c8c02e08e98b69379db92c3bb2d9ce7df384226657" Jul 17 18:59:26.277941 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:26.277902 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs"] Jul 17 18:59:26.287356 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:26.287330 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02bfc37f-62zqs"] Jul 17 18:59:27.001965 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:27.001939 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nk6r6_ead021cb-e13c-433e-b94f-de93c3165044/serve-healthcheck-canary/0.log" Jul 17 18:59:27.224874 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:27.224795 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cb9f3a2-74bf-45f5-9945-75ffdd80a501" path="/var/lib/kubelet/pods/4cb9f3a2-74bf-45f5-9945-75ffdd80a501/volumes" Jul 17 18:59:27.837118 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:27.837093 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g6q7x_76665724-fb0c-43fe-b09d-e3d43ede2fc0/kube-rbac-proxy/0.log" Jul 17 18:59:27.869503 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:27.869467 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g6q7x_76665724-fb0c-43fe-b09d-e3d43ede2fc0/exporter/0.log" Jul 17 18:59:27.894673 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:27.894648 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g6q7x_76665724-fb0c-43fe-b09d-e3d43ede2fc0/extractor/0.log" Jul 17 18:59:28.813107 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:28.813070 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-7rf9z_aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76/discovery/0.log" Jul 17 18:59:28.849463 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:28.849441 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84bfff449b-t7wpv_23012aad-30af-44b0-af8e-f341abe9d845/kube-auth-proxy/0.log" Jul 17 18:59:30.301802 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:30.301773 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-7rf9z_aa7d4a7f-c6d2-4b1e-808d-a1e44b37bd76/discovery/0.log" Jul 17 18:59:30.340531 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:30.340498 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84bfff449b-t7wpv_23012aad-30af-44b0-af8e-f341abe9d845/kube-auth-proxy/0.log" Jul 17 18:59:31.380286 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:31.380258 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d8bdcf84-f82c8_378e4b36-e952-4175-80f4-5756ab57c91a/manager/0.log" Jul 17 18:59:31.404804 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:31.404780 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d8bdcf84-f82c8_378e4b36-e952-4175-80f4-5756ab57c91a/copy-manifests/0.log" Jul 17 18:59:32.544773 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:32.544727 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-f76c48b9d-9gpn8_27b397ee-fdd1-43d2-8439-7396211b1acd/manager/0.log" Jul 17 18:59:32.595086 ip-10-0-132-62 kubenswrapper[2572]: I0717 18:59:32.595055 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-fvgr4_c4772f6a-5579-43aa-848b-e5871d4709dc/openshift-lws-operator/0.log"