Jul 17 12:22:04.860072 ip-10-0-131-210 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 12:22:05.303529 ip-10-0-131-210 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 12:22:05.303529 ip-10-0-131-210 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 12:22:05.303529 ip-10-0-131-210 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 12:22:05.303529 ip-10-0-131-210 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 12:22:05.303529 ip-10-0-131-210 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 12:22:05.303529 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.299402    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306895    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306915    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306919    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306923    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306925    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306928    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306931    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306934    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306939    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306943    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306947    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306951    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306954    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306957    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306960    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306962    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306965    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306968    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 12:22:05.319560 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306970    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306973    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306975    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306978    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306980    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306983    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306986    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306988    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.306999    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307002    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307005    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307008    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307010    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307012    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307015    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307017    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307020    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307022    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307025    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307028    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 12:22:05.321971 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307031    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307034    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307038    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307040    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307044    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307048    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307050    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307053    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307056    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307058    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307061    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307064    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307067    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307069    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307072    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307075    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307078    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307081    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307084    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307086    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 12:22:05.329811 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307089    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307091    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307095    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307097    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307099    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307102    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307105    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307107    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307110    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307113    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307117    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307121    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307124    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307128    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307131    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307134    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307136    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307139    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307142    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 12:22:05.331978 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307144    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307148    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307151    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307154    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307156    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307160    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307162    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307165    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307168    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307170    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307172    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307175    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307178    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307182    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307186    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307189    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307191    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307194    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307197    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 12:22:05.333694 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307199    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307202    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307205    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307976    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307984    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307989    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307993    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307996    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.307999    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308002    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308004    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308007    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308009    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308012    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308014    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308017    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308020    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308023    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308026    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308029    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 12:22:05.335965 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308031    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308034    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308036    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308039    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308042    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308044    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308047    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308049    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308051    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308054    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308056    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308059    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308062    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308065    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308067    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308070    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308072    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308075    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308077    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308080    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 12:22:05.349788 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308083    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308086    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308088    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308091    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308093    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308096    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308099    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308101    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308104    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308107    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308110    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308112    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308115    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308118    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308120    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308122    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308125    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308128    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308131    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 12:22:05.352102 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308133    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308136    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308138    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308140    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308143    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308145    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308148    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308150    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308153    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308156    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308159    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308161    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308163    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308166    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308168    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308172    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308177    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308183    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308186    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 12:22:05.354018 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308190    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308193    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308196    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308198    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308202    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308204    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308207    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308210    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308213    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308216    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308219    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308221    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308223    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308226    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308229    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308231    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308234    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308237    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308239    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308242    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 12:22:05.356367 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308244    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308246    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308249    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308252    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308659    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308666    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308669    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308672    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308675    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308678    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308680    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308683    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308686    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308689    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308691    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308694    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308697    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308700    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308703    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308710    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 12:22:05.388710 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308713    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 12:22:05.381838 ip-10-0-131-210 systemd[1]: Started Kubernetes Kubelet.
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308716    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308718    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308721    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308724    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308726    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308729    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308732    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308735    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308737    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308741    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308745    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308768    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308772    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308775    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308777    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308780    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308783    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308785    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308788    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 12:22:05.390577 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308790    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308793    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308796    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308799    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308802    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308804    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308807    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308809    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308812    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308815    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308818    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308823    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308826    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308828    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308831    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308834    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308836    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308839    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308841    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 12:22:05.618019 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308844    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308846    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308849    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308852    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308854    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308856    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308859    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308862    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308864    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308867    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308869    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308872    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308874    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308878    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308881    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308885    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308888    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308891    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308894    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308896    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 12:22:05.632782 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308899    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308903    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308905    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308908    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308914    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308917    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308919    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308922    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308924    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308927    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308929    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308932    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308934    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308937    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308939    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308942    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308944    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308947    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308949    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308952    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 12:22:05.634830 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308955    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308957    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308960    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.308963    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309042    2569 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309049    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309056    2569 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309061    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309066    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309069    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309073    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309078    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309081    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309085    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309088    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309091    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309097    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309101    2569 flags.go:64] FLAG: --cgroup-root=""
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309104    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309106    2569 flags.go:64] FLAG: --client-ca-file=""
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309110    2569 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309113    2569 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309117    2569 flags.go:64] FLAG: --cluster-domain=""
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309120    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 12:22:05.635931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309123    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309126    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309129    2569 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309132    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309135    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309139    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309142    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309145    2569 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309148    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309151    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309154    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309157    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309161    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309164    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309167    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309170    2569 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309173    2569 flags.go:64] FLAG: --enable-server="true"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309176    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309181    2569 flags.go:64] FLAG: --event-burst="100"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309184    2569 flags.go:64] FLAG: --event-qps="50"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309186    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309189    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309192    2569 flags.go:64] FLAG: --eviction-hard=""
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309196    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 12:22:05.637269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309201    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309205    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309209    2569 flags.go:64] FLAG: --eviction-soft=""
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309211    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309214    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309217    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309220    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309223    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309226    2569 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309228    2569 flags.go:64] FLAG: --feature-gates=""
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309232    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309235    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309238    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309241    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309244    2569 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309247    2569 flags.go:64] FLAG: --help="false"
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309250    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-131-210.ec2.internal"
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309253    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309256    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309259    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309263    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309266    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309270    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 12:22:05.638908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309272    2569 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309275    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309278    2569 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309281    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309284    2569 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309286    2569 flags.go:64] FLAG: --kube-reserved=""
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309289    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309292    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309295    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309300    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309303    2569 flags.go:64] FLAG: --lock-file=""
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309306    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309309    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309312    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309317    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309320    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309323    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309325    2569 flags.go:64] FLAG: --logging-format="text"
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309328    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309332    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309334    2569 flags.go:64] FLAG: --manifest-url=""
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309337    2569 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309341    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309344    2569 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309348    2569 flags.go:64] FLAG: --max-pods="110"
Jul 17 12:22:05.640493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309351    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309354    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309357    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309360    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309363    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309366    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309369    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309377    2569 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309380    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309383    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309386    2569 flags.go:64] FLAG: --pod-cidr=""
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309388    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309394    2569 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309397    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309400    2569 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309403    2569 flags.go:64] FLAG: --port="10250"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309407    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309411    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07eb50c1947fe08b6"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309414    2569 flags.go:64] FLAG: --qos-reserved=""
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309417    2569 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309420    2569 flags.go:64] FLAG: --register-node="true"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309423    2569 flags.go:64] FLAG: --register-with-taints=""
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309427    2569 flags.go:64] FLAG: --registry-burst="10"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309430    2569 flags.go:64] FLAG: --registry-qps="5"
Jul 17 12:22:05.641616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309433    2569 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309436    2569 flags.go:64] FLAG: --reserved-memory=""
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309440    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309443    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309446    2569 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309449    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309452    2569 flags.go:64] FLAG: --runonce="false"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309455    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309458    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309461    2569 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309464    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309467    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309470    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309473    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309476    2569 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309479    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309481    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309484    2569 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309487    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309491    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309493    2569 flags.go:64] FLAG: --system-cgroups=""
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309496    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309501    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309504    2569 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309508    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 12:22:05.642362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309513    2569 flags.go:64] FLAG: --tls-min-version=""
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309516    2569 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309519    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309523    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309526    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309529    2569 flags.go:64] FLAG: --v="2"
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309533    2569 flags.go:64] FLAG: --version="false"
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309537    2569 flags.go:64] FLAG: --vmodule=""
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309541    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309544    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309660    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309663    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309667    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309671    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309675    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309678    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309681    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309684    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309687    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309689    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309692    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309695    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 12:22:05.643459 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309697    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309700    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309702    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309705    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309707    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309710    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309712    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309715    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309717    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309721    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309726    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309730    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309733    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309736    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309739    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309742    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309745    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309761    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309765    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 12:22:05.644505 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309768    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309771    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309774    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309777    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309780    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309782    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309785    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309788    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309790    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309793    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309796    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309798    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309802    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309804    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309807    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309810    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309812    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309815    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309817    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309820    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 12:22:05.645252 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309822    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309825    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309830    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309834    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309836    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309839    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309841    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309844    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309847    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309850    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309853    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309855    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309858    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309860    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309863    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309866    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309868    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309871    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309873    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309875    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 12:22:05.646010 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309878    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309881    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309883    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309885    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309888    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309891    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309894    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309896    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309898    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309900    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309903    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309905    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309908    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309910    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309914    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309918    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309921    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309923    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309926    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309930    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 12:22:05.646738 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309933    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 12:22:05.647481 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309936    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 12:22:05.647481 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309939    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 12:22:05.647481 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309942    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 12:22:05.647481 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309945    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 12:22:05.647481 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309947    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 12:22:05.647481 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309950    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 12:22:05.647481 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.309953    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 12:22:05.647481 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.309962    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 12:22:05.647481 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.316951    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 12:22:05.647481 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.316970    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 12:22:05.647481 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.316992    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 12:22:05.647481 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.317001    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 12:22:05.647481 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317059    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 12:22:05.647481 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317064    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 12:22:05.647481 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317068    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317071    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317075    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317078    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317081    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317084    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317088    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317090    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317094    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317096    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317099    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317102    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317105    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317108    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317110    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317113    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317116    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317119    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317122    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317126    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 12:22:05.648255 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317129    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317132    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317134    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317137    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317140    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317142    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317145    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317147    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317150    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317152    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317155    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317158    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317160    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317163    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317166    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317169    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317172    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317174    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317177    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317180    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 12:22:05.649005 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317182    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317185    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317189    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317193    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317196    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317199    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317202    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317207    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317210    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317214    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317218    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317222    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317226    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317229    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317232    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317235    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317237    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317240    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317243    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 12:22:05.649742 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317246    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317248    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317251    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317253    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317256    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317258    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317261    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317263    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317266    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317269    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317271    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317273    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317276    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317278    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317282    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317286    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317288    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317291    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317294    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317298    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 12:22:05.650498 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317302    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 12:22:05.651257 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317305    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 12:22:05.651257 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317309    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 12:22:05.651257 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317312    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 12:22:05.651257 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317315    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 12:22:05.651257 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317318    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 12:22:05.651257 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317320    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 12:22:05.651257 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317323    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 12:22:05.651257 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317325    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 12:22:05.651257 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317328    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 12:22:05.651257 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317331    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 12:22:05.651257 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317333    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 12:22:05.651257 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317336    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 12:22:05.651257 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317338    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 12:22:05.651257 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317341    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 12:22:05.651257 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317343    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 12:22:05.651257 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317346    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 12:22:05.651257 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317348    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 12:22:05.657149 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.317354    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 12:22:05.657149 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317474    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 12:22:05.657149 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317479    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 12:22:05.657149 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317482    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 12:22:05.657149 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317484    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 12:22:05.657149 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317488    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 12:22:05.657149 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317490    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 12:22:05.657149 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317493    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 12:22:05.657149 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317495    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 12:22:05.657149 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317498    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 12:22:05.657149 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317500    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 12:22:05.657149 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317503    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 12:22:05.657149 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317505    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 12:22:05.657149 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317508    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317510    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317514    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317518    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317521    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317523    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317526    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317529    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317531    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317534    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317537    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317539    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317542    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317544    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317547    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317549    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317552    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317554    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317557    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317559    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 12:22:05.657928 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317561    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317564    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317566    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317569    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317572    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317575    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317577    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317579    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317582    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317584    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317587    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317589    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317591    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317594    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317598    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317604    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317607    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317609    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317612    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317614    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 12:22:05.754607 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317618    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317620    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317623    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317625    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317628    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317630    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317633    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317636    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317638    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317641    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317644    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317648    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317651    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317653    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317656    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317659    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317662    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317665    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317667    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 12:22:05.755350 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317670    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317673    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317676    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317679    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317682    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317684    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317687    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317690    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317694    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317697    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317700    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317702    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317705    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317707    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317710    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317713    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317715    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317718    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317720    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 12:22:05.756070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317723    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 12:22:05.756826 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317725    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 12:22:05.756826 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317728    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 12:22:05.756826 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317730    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 12:22:05.756826 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317733    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 12:22:05.756826 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317735    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 12:22:05.756826 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317739    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 12:22:05.756826 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317742    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 12:22:05.756826 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:05.317745    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 12:22:05.756826 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.317768    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 12:22:05.756826 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.317897    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 12:22:05.756826 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.320150    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 12:22:05.756826 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.321659    2569 server.go:1019] "Starting client certificate rotation"
Jul 17 12:22:05.756826 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.322008    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 12:22:05.756826 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.322045    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 12:22:05.757575 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.325997    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 12:22:05.757575 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.328173    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 12:22:05.757575 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.344145    2569 log.go:25] "Validated CRI v1 runtime API"
Jul 17 12:22:05.757575 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.346433    2569 log.go:25] "Validated CRI v1 image API"
Jul 17 12:22:05.757575 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.347249    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 12:22:05.757575 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.351118    2569 fs.go:135] Filesystem UUIDs: map[66307df8-14f8-4463-a09b-c26c5805f2e1:/dev/nvme0n1p4 67270094-5932-47cc-ad05-37e6e99cd2c4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 17 12:22:05.757575 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.351142    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 12:22:05.758302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.356412    2569 manager.go:216] Machine: {Timestamp:2026-07-17 12:22:05.354993381 +0000 UTC m=+0.359040273 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100021 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec296b2118da9718fd73ed90b6faab3e SystemUUID:ec296b21-18da-9718-fd73-ed90b6faab3e BootID:fcb64371-9407-4805-9ac1-8a892cc46753 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:af:da:2b:a2:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:af:da:2b:a2:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:96:c5:e5:49:1d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 12:22:05.758302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.356520    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 12:22:05.758302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.356620    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 12:22:05.758302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.358931    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 12:22:05.758302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.360166    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 12:22:05.758302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.360196    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-210.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 12:22:05.758302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.360958    2569 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 12:22:05.758302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.360969    2569 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 12:22:05.758302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.360986    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 12:22:05.758302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.361003    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 12:22:05.758302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.362287    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 12:22:05.758302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.364898    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 12:22:05.758302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.365019    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 12:22:05.758302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.365060    2569 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 12:22:05.758302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.365076    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 12:22:05.758302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.365090    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 12:22:05.758302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.365102    2569 kubelet.go:398] "Adding apiserver pod source"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.365113    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.365470    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.365486    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.367608    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.369993    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.370363    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.372362    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.372379    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.372387    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.372393    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.372407    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.372419    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.372428    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.372434    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.372461    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.372472    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.372846    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.372861    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.373897    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.373910    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:05.377189    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.377247    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-210.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:05.377396    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.380785    2569 server.go:1268] "Started kubelet"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.380866    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.380871    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.380946    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.383055    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.383296    2569 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:05.388909    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-210.ec2.internal.18c313249f459800  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-210.ec2.internal,UID:ip-10-0-131-210.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-210.ec2.internal,},FirstTimestamp:2026-07-17 12:22:05.380737024 +0000 UTC m=+0.384783950,LastTimestamp:2026-07-17 12:22:05.380737024 +0000 UTC m=+0.384783950,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-210.ec2.internal,}"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.390636    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.390635    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:05.390856    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.391261    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.391268    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.391301    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.391453    2569 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.391463    2569 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 12:22:05.759014 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:05.406836    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:05.406849    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.406870    2569 factory.go:55] Registering systemd factory
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.406885    2569 factory.go:223] Registration of the systemd container factory successfully
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:05.406948    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.407154    2569 factory.go:153] Registering CRI-O factory
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.407166    2569 factory.go:223] Registration of the crio container factory successfully
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.407222    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.407246    2569 factory.go:103] Registering Raw factory
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.407260    2569 manager.go:1196] Started watching for new ooms in manager
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.407702    2569 manager.go:349] Starting recovery of all containers
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.408277    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b4rwd"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.409140    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b4rwd"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.426118    2569 manager.go:354] Recovery completed
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.430438    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.432654    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.432696    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.432707    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.433352    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.433361    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.433381    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.433389    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.434129    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.434160    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.434174    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.434523    2569 policy_none.go:49] "None policy: Start"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.434538    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.434550    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.435698    2569 policy_none.go:47] "Start"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:05.491112    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.495500    2569 manager.go:337] "Starting Device Plugin manager"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:05.495534    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.495544    2569 server.go:87] "Starting device plugin registration server"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.495879    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.495892    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.496095    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.496191    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.496202    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:05.496597    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:05.496651    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.596923    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.598063    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.598095    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.598108    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.598133    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-210.ec2.internal"
Jul 17 12:22:05.760087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.607948    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-210.ec2.internal"
Jul 17 12:22:05.761493 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:05.607973    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-210.ec2.internal\": node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 17 12:22:05.761493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.626331    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 12:22:05.761493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.627589    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 12:22:05.761493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.627632    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 12:22:05.761493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.627663    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 12:22:05.761493 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:05.627706    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 17 12:22:05.761493 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:05.628624    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 17 12:22:05.761493 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:05.629720    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 12:22:06.673352 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.672888    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 12:22:06.673352 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:06.672897    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-210.ec2.internal\" not found"
Jul 17 12:22:06.673352 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.672911    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 12:17:05 +0000 UTC" deadline="2028-02-29 03:39:30.752583358 +0000 UTC"
Jul 17 12:22:06.673352 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.673131    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14199h17m24.079458418s"
Jul 17 12:22:06.673352 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.672950    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal"]
Jul 17 12:22:06.673352 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.673245    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.674695    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.674720    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.674729    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.676037    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.676389    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.676415    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.682922    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.684760    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.689049    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.690817    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.693149    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.693179    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.693193    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.693200    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.693214    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.693224    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.695017    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.701842    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.702678    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.709864    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.732016    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.737485    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.741775    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:06.741812    2569 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.743239    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 12:22:06.751396 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:06.743272    2569 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-210.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:06.859423 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.773472    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d412a8e9141d4c1c1a722c1bb3d88144-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal\" (UID: \"d412a8e9141d4c1c1a722c1bb3d88144\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:06.859423 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.773503    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d412a8e9141d4c1c1a722c1bb3d88144-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal\" (UID: \"d412a8e9141d4c1c1a722c1bb3d88144\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:06.859423 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.773533    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc30aca19fb5f39f59f0d9bb7d486e95-config\") pod \"kube-apiserver-proxy-ip-10-0-131-210.ec2.internal\" (UID: \"bc30aca19fb5f39f59f0d9bb7d486e95\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:06.943994 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.873992    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d412a8e9141d4c1c1a722c1bb3d88144-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal\" (UID: \"d412a8e9141d4c1c1a722c1bb3d88144\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:06.943994 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.874034    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d412a8e9141d4c1c1a722c1bb3d88144-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal\" (UID: \"d412a8e9141d4c1c1a722c1bb3d88144\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:06.943994 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.874048    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc30aca19fb5f39f59f0d9bb7d486e95-config\") pod \"kube-apiserver-proxy-ip-10-0-131-210.ec2.internal\" (UID: \"bc30aca19fb5f39f59f0d9bb7d486e95\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:06.943994 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.874086    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc30aca19fb5f39f59f0d9bb7d486e95-config\") pod \"kube-apiserver-proxy-ip-10-0-131-210.ec2.internal\" (UID: \"bc30aca19fb5f39f59f0d9bb7d486e95\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:06.943994 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.874114    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d412a8e9141d4c1c1a722c1bb3d88144-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal\" (UID: \"d412a8e9141d4c1c1a722c1bb3d88144\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:06.943994 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:06.874124    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d412a8e9141d4c1c1a722c1bb3d88144-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal\" (UID: \"d412a8e9141d4c1c1a722c1bb3d88144\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:07.044143 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.044061    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:07.046273 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.046244    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal"
Jul 17 12:22:07.540797 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:07.540761    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd412a8e9141d4c1c1a722c1bb3d88144.slice/crio-e7c2377f162c46d8292ebcc35a0b46cb67e9b23a81c88e33b39b4520d4737cac WatchSource:0}: Error finding container e7c2377f162c46d8292ebcc35a0b46cb67e9b23a81c88e33b39b4520d4737cac: Status 404 returned error can't find the container with id e7c2377f162c46d8292ebcc35a0b46cb67e9b23a81c88e33b39b4520d4737cac
Jul 17 12:22:07.543589 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:07.543565    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc30aca19fb5f39f59f0d9bb7d486e95.slice/crio-2edf616d0696c5519c0a97a9187dc5f7cd49c38635b9942aabe28e1b0e1e50a9 WatchSource:0}: Error finding container 2edf616d0696c5519c0a97a9187dc5f7cd49c38635b9942aabe28e1b0e1e50a9: Status 404 returned error can't find the container with id 2edf616d0696c5519c0a97a9187dc5f7cd49c38635b9942aabe28e1b0e1e50a9
Jul 17 12:22:07.545359 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.545345    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 12:22:07.673256 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.673224    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 12:22:07.673256 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.673238    2569 apiserver.go:52] "Watching apiserver"
Jul 17 12:22:07.676446 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.676406    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal" event={"ID":"bc30aca19fb5f39f59f0d9bb7d486e95","Type":"ContainerStarted","Data":"2edf616d0696c5519c0a97a9187dc5f7cd49c38635b9942aabe28e1b0e1e50a9"}
Jul 17 12:22:07.677527 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.677507    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal" event={"ID":"d412a8e9141d4c1c1a722c1bb3d88144","Type":"ContainerStarted","Data":"e7c2377f162c46d8292ebcc35a0b46cb67e9b23a81c88e33b39b4520d4737cac"}
Jul 17 12:22:07.681360 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.681333    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 12:22:07.681785 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.681766    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-xp78t","openshift-ovn-kubernetes/ovnkube-node-27xxg","kube-system/global-pull-secret-syncer-nw9sw","kube-system/konnectivity-agent-c2c46","kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal","openshift-image-registry/node-ca-q7w4l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal","openshift-multus/multus-additional-cni-plugins-whc8l","openshift-multus/multus-g4c6m","openshift-network-diagnostics/network-check-target-hcnj4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g","openshift-cluster-node-tuning-operator/tuned-f7cdh","openshift-dns/node-resolver-lmgzk","openshift-multus/network-metrics-daemon-4nbwg"]
Jul 17 12:22:07.683402 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.683385    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 12:22:07.683648 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.683635    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:07.683707 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:07.683692    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hcnj4" podUID="db4fbe66-76bb-480e-9a7f-893b764ec3bf"
Jul 17 12:22:07.684593 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.684581    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.685683 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.685668    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:07.685769 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:07.685716    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nw9sw" podUID="a4b940c7-3fe3-4290-b253-efde9655c166"
Jul 17 12:22:07.686816 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.686797    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c2c46"
Jul 17 12:22:07.687018 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.686981    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2zvf4\""
Jul 17 12:22:07.687083 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.687031    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 12:22:07.687083 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.687042    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 12:22:07.687174 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.687088    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 12:22:07.687782 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.687766    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:07.687860 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:07.687837    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:22:07.688818 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.688804    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q7w4l"
Jul 17 12:22:07.689147 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.689129    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cf6hz\""
Jul 17 12:22:07.689257 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.689231    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 12:22:07.689322 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.689308    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 12:22:07.690022 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.690007    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.692415 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.691425    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 12:22:07.692415 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.691633    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lmgzk"
Jul 17 12:22:07.692415 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.692108    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w54p6\""
Jul 17 12:22:07.692573 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.692545    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 12:22:07.692619 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.692609    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 12:22:07.692655 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.692642    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 12:22:07.694570 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.693162    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 12:22:07.694570 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.693176    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 12:22:07.694570 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.693241    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7xkpc\""
Jul 17 12:22:07.694570 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.693171    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 12:22:07.694570 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.694286    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 12:22:07.695083 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.695063    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 12:22:07.695173 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.695159    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rfzz6\""
Jul 17 12:22:07.695436 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.695417    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.696894 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.696877    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xp78t"
Jul 17 12:22:07.698129 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.697956    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 12:22:07.698129 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.697971    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 12:22:07.698129 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.698024    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 12:22:07.698129 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.698062    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 12:22:07.698349 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.698176    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 12:22:07.698349 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.698189    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bpjc7\""
Jul 17 12:22:07.698349 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.698192    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.698937 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.698921    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 12:22:07.699026 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.698998    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 12:22:07.699026 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.699019    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dxvln\""
Jul 17 12:22:07.699115 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.699019    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 12:22:07.699254 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.699238    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 12:22:07.699447 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.699434    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.700432 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.700391    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 12:22:07.700505 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.700435    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cpk4q\""
Jul 17 12:22:07.700505 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.700458    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 12:22:07.701684 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.701666    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 12:22:07.701868 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.701855    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qfspz\""
Jul 17 12:22:07.702128 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.702115    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 12:22:07.703922 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.703830    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q98r7"
Jul 17 12:22:07.708967 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.708951    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q98r7"
Jul 17 12:22:07.777657 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.777627    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-run-ovn-kubernetes\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.777657 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.777655    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-os-release\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.777923 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.777686    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-host-run-k8s-cni-cncf-io\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.777923 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.777707    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-host-run-netns\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.777923 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.777729    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/79ba2047-ca88-49e0-8b8e-621b724dd8d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.777923 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.777782    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-ovnkube-config\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.777923 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.777831    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-ovnkube-script-lib\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.777923 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.777868    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn4rf\" (UniqueName: \"kubernetes.io/projected/a2bedbc3-150b-4334-a07f-574108c2901a-kube-api-access-wn4rf\") pod \"node-ca-q7w4l\" (UID: \"a2bedbc3-150b-4334-a07f-574108c2901a\") " pod="openshift-image-registry/node-ca-q7w4l"
Jul 17 12:22:07.777923 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.777886    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbxdw\" (UniqueName: \"kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw\") pod \"network-check-target-hcnj4\" (UID: \"db4fbe66-76bb-480e-9a7f-893b764ec3bf\") " pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:07.777923 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.777900    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-slash\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.777923 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.777913    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-system-cni-dir\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.777923 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.777925    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80c7a3a0-1833-46d5-bd61-7843b21bd530-cni-binary-copy\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.778345 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.777939    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/79ba2047-ca88-49e0-8b8e-621b724dd8d0-device-dir\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.778345 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.777952    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqp9z\" (UniqueName: \"kubernetes.io/projected/79ba2047-ca88-49e0-8b8e-621b724dd8d0-kube-api-access-gqp9z\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.778345 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778002    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-run-netns\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.778345 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778030    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt997\" (UniqueName: \"kubernetes.io/projected/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-kube-api-access-jt997\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.778345 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778054    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/79ba2047-ca88-49e0-8b8e-621b724dd8d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.778345 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778086    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9916c7a-cfa6-4b17-bea4-0642e4dce23e-agent-certs\") pod \"konnectivity-agent-c2c46\" (UID: \"a9916c7a-cfa6-4b17-bea4-0642e4dce23e\") " pod="kube-system/konnectivity-agent-c2c46"
Jul 17 12:22:07.778345 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778106    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs\") pod \"network-metrics-daemon-4nbwg\" (UID: \"966fc2f5-1448-4a14-8b27-21e714628e1a\") " pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:07.778345 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778128    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/460481eb-fc51-49b6-b8f7-7e892d0d0e64-tmp-dir\") pod \"node-resolver-lmgzk\" (UID: \"460481eb-fc51-49b6-b8f7-7e892d0d0e64\") " pod="openshift-dns/node-resolver-lmgzk"
Jul 17 12:22:07.778345 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778149    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-cni-netd\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.778345 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778180    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-cnibin\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.778345 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778202    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-multus-socket-dir-parent\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.778345 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778216    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-host-var-lib-cni-bin\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.778345 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778229    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/79ba2047-ca88-49e0-8b8e-621b724dd8d0-socket-dir\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.778345 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778266    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/79ba2047-ca88-49e0-8b8e-621b724dd8d0-registration-dir\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.778345 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778292    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-etc-openvswitch\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.778345 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778329    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-run-ovn\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.778979 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778352    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-cni-bin\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.778979 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778389    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-node-log\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.778979 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778410    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-log-socket\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.778979 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778432    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret\") pod \"global-pull-secret-syncer-nw9sw\" (UID: \"a4b940c7-3fe3-4290-b253-efde9655c166\") " pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:07.778979 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778453    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-multus-cni-dir\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.778979 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778474    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-multus-conf-dir\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.778979 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778495    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-kubelet\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.778979 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778517    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-ovn-node-metrics-cert\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.778979 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778541    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a4b940c7-3fe3-4290-b253-efde9655c166-kubelet-config\") pod \"global-pull-secret-syncer-nw9sw\" (UID: \"a4b940c7-3fe3-4290-b253-efde9655c166\") " pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:07.778979 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778565    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2bedbc3-150b-4334-a07f-574108c2901a-host\") pod \"node-ca-q7w4l\" (UID: \"a2bedbc3-150b-4334-a07f-574108c2901a\") " pod="openshift-image-registry/node-ca-q7w4l"
Jul 17 12:22:07.778979 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778585    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-etc-kubernetes\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.778979 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778606    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2rnf\" (UniqueName: \"kubernetes.io/projected/80c7a3a0-1833-46d5-bd61-7843b21bd530-kube-api-access-q2rnf\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.778979 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778632    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9916c7a-cfa6-4b17-bea4-0642e4dce23e-konnectivity-ca\") pod \"konnectivity-agent-c2c46\" (UID: \"a9916c7a-cfa6-4b17-bea4-0642e4dce23e\") " pod="kube-system/konnectivity-agent-c2c46"
Jul 17 12:22:07.778979 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778654    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpp7c\" (UniqueName: \"kubernetes.io/projected/460481eb-fc51-49b6-b8f7-7e892d0d0e64-kube-api-access-gpp7c\") pod \"node-resolver-lmgzk\" (UID: \"460481eb-fc51-49b6-b8f7-7e892d0d0e64\") " pod="openshift-dns/node-resolver-lmgzk"
Jul 17 12:22:07.778979 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778674    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2bedbc3-150b-4334-a07f-574108c2901a-serviceca\") pod \"node-ca-q7w4l\" (UID: \"a2bedbc3-150b-4334-a07f-574108c2901a\") " pod="openshift-image-registry/node-ca-q7w4l"
Jul 17 12:22:07.778979 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778697    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-host-var-lib-cni-multus\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.778979 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778729    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-hostroot\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.779408 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778772    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/80c7a3a0-1833-46d5-bd61-7843b21bd530-multus-daemon-config\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.779408 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778794    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-var-lib-openvswitch\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.779408 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778820    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-host-run-multus-certs\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.779408 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778843    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/79ba2047-ca88-49e0-8b8e-621b724dd8d0-sys-fs\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.779408 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778858    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-env-overrides\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.779408 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778871    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lvsm\" (UniqueName: \"kubernetes.io/projected/966fc2f5-1448-4a14-8b27-21e714628e1a-kube-api-access-7lvsm\") pod \"network-metrics-daemon-4nbwg\" (UID: \"966fc2f5-1448-4a14-8b27-21e714628e1a\") " pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:07.779408 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778885    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-run-openvswitch\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.779408 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778899    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.779408 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778914    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a4b940c7-3fe3-4290-b253-efde9655c166-dbus\") pod \"global-pull-secret-syncer-nw9sw\" (UID: \"a4b940c7-3fe3-4290-b253-efde9655c166\") " pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:07.779408 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778926    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-host-var-lib-kubelet\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.779408 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778953    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-run-systemd\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.779408 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778976    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/460481eb-fc51-49b6-b8f7-7e892d0d0e64-hosts-file\") pod \"node-resolver-lmgzk\" (UID: \"460481eb-fc51-49b6-b8f7-7e892d0d0e64\") " pod="openshift-dns/node-resolver-lmgzk"
Jul 17 12:22:07.779408 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.778989    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-systemd-units\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.791956 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.791931    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 12:22:07.879236 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879200    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2bedbc3-150b-4334-a07f-574108c2901a-serviceca\") pod \"node-ca-q7w4l\" (UID: \"a2bedbc3-150b-4334-a07f-574108c2901a\") " pod="openshift-image-registry/node-ca-q7w4l"
Jul 17 12:22:07.879236 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879234    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-host-var-lib-cni-multus\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.879429 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879256    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-hostroot\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.879429 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879273    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/80c7a3a0-1833-46d5-bd61-7843b21bd530-multus-daemon-config\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.879429 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879292    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.879429 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879307    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-kubernetes\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.879429 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879329    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-var-lib-openvswitch\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.879429 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879350    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-host-run-multus-certs\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.879429 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879348    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-host-var-lib-cni-multus\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.879429 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879371    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.879429 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879391    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-sysctl-d\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.879429 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879407    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-var-lib-openvswitch\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.879429 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879407    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-hostroot\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.879429 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879414    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-sysctl-conf\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.880061 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879443    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-lib-modules\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.880061 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879446    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-host-run-multus-certs\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.880061 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879485    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-tuned\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.880061 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879516    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/79ba2047-ca88-49e0-8b8e-621b724dd8d0-sys-fs\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.880061 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879579    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/79ba2047-ca88-49e0-8b8e-621b724dd8d0-sys-fs\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.880061 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879580    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-env-overrides\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.880061 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879607    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-host\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.880061 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879630    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lvsm\" (UniqueName: \"kubernetes.io/projected/966fc2f5-1448-4a14-8b27-21e714628e1a-kube-api-access-7lvsm\") pod \"network-metrics-daemon-4nbwg\" (UID: \"966fc2f5-1448-4a14-8b27-21e714628e1a\") " pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:07.880061 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879654    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-run-openvswitch\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.880061 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879689    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-run-openvswitch\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.880061 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879717    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.880061 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879763    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a4b940c7-3fe3-4290-b253-efde9655c166-dbus\") pod \"global-pull-secret-syncer-nw9sw\" (UID: \"a4b940c7-3fe3-4290-b253-efde9655c166\") " pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:07.880061 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879807    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-sys\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.880061 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879808    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2bedbc3-150b-4334-a07f-574108c2901a-serviceca\") pod \"node-ca-q7w4l\" (UID: \"a2bedbc3-150b-4334-a07f-574108c2901a\") " pod="openshift-image-registry/node-ca-q7w4l"
Jul 17 12:22:07.880061 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879833    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.880061 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879835    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-var-lib-kubelet\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.880061 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879895    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-host-var-lib-kubelet\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.880518 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879903    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/80c7a3a0-1833-46d5-bd61-7843b21bd530-multus-daemon-config\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.880518 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879932    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-run-systemd\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.880518 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879950    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-host-var-lib-kubelet\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.880518 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879974    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-tmp\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.880518 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879992    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-run-systemd\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.880518 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879997    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-env-overrides\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.880518 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.879996    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/460481eb-fc51-49b6-b8f7-7e892d0d0e64-hosts-file\") pod \"node-resolver-lmgzk\" (UID: \"460481eb-fc51-49b6-b8f7-7e892d0d0e64\") " pod="openshift-dns/node-resolver-lmgzk"
Jul 17 12:22:07.880518 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880033    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/460481eb-fc51-49b6-b8f7-7e892d0d0e64-hosts-file\") pod \"node-resolver-lmgzk\" (UID: \"460481eb-fc51-49b6-b8f7-7e892d0d0e64\") " pod="openshift-dns/node-resolver-lmgzk"
Jul 17 12:22:07.880518 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880038    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-systemd-units\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.880518 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880046    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a4b940c7-3fe3-4290-b253-efde9655c166-dbus\") pod \"global-pull-secret-syncer-nw9sw\" (UID: \"a4b940c7-3fe3-4290-b253-efde9655c166\") " pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:07.880518 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880062    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm4rd\" (UniqueName: \"kubernetes.io/projected/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-kube-api-access-hm4rd\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.880518 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880064    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-systemd-units\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.880518 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880082    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-modprobe-d\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.880518 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880099    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-run-ovn-kubernetes\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.880518 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880133    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6cgp\" (UniqueName: \"kubernetes.io/projected/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-kube-api-access-m6cgp\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.880518 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880168    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-os-release\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.880518 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880140    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-run-ovn-kubernetes\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.881298 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880184    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-host-run-k8s-cni-cncf-io\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.881298 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880203    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-host-run-netns\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.881298 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880211    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-host-run-k8s-cni-cncf-io\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.881298 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880221    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/298d5722-f081-490f-bc9c-dd2cd7bfba23-iptables-alerter-script\") pod \"iptables-alerter-xp78t\" (UID: \"298d5722-f081-490f-bc9c-dd2cd7bfba23\") " pod="openshift-network-operator/iptables-alerter-xp78t"
Jul 17 12:22:07.881298 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880235    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-host-run-netns\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.881298 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880246    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/79ba2047-ca88-49e0-8b8e-621b724dd8d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.881298 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880252    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-os-release\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.881298 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880261    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-ovnkube-config\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.881298 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880279    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/79ba2047-ca88-49e0-8b8e-621b724dd8d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.881298 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880281    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-ovnkube-script-lib\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.881298 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880335    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn4rf\" (UniqueName: \"kubernetes.io/projected/a2bedbc3-150b-4334-a07f-574108c2901a-kube-api-access-wn4rf\") pod \"node-ca-q7w4l\" (UID: \"a2bedbc3-150b-4334-a07f-574108c2901a\") " pod="openshift-image-registry/node-ca-q7w4l"
Jul 17 12:22:07.881298 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880357    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/298d5722-f081-490f-bc9c-dd2cd7bfba23-host-slash\") pod \"iptables-alerter-xp78t\" (UID: \"298d5722-f081-490f-bc9c-dd2cd7bfba23\") " pod="openshift-network-operator/iptables-alerter-xp78t"
Jul 17 12:22:07.881298 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880405    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.881298 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880428    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-sysconfig\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.881298 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880453    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbxdw\" (UniqueName: \"kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw\") pod \"network-check-target-hcnj4\" (UID: \"db4fbe66-76bb-480e-9a7f-893b764ec3bf\") " pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:07.881298 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880474    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-slash\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.881298 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880495    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-system-cni-dir\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.881915 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880518    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80c7a3a0-1833-46d5-bd61-7843b21bd530-cni-binary-copy\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.881915 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880530    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-slash\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.881915 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880572    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-system-cni-dir\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.881915 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880615    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-run\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.881915 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880643    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/79ba2047-ca88-49e0-8b8e-621b724dd8d0-device-dir\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.881915 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880670    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqp9z\" (UniqueName: \"kubernetes.io/projected/79ba2047-ca88-49e0-8b8e-621b724dd8d0-kube-api-access-gqp9z\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.881915 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880693    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-run-netns\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.881915 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880715    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt997\" (UniqueName: \"kubernetes.io/projected/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-kube-api-access-jt997\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.881915 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880716    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-ovnkube-config\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.881915 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880744    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/79ba2047-ca88-49e0-8b8e-621b724dd8d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.881915 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880740    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/79ba2047-ca88-49e0-8b8e-621b724dd8d0-device-dir\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.881915 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880775    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-ovnkube-script-lib\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.881915 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880782    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-run-netns\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.881915 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880845    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/79ba2047-ca88-49e0-8b8e-621b724dd8d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.881915 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880867    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9916c7a-cfa6-4b17-bea4-0642e4dce23e-agent-certs\") pod \"konnectivity-agent-c2c46\" (UID: \"a9916c7a-cfa6-4b17-bea4-0642e4dce23e\") " pod="kube-system/konnectivity-agent-c2c46"
Jul 17 12:22:07.881915 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880898    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs\") pod \"network-metrics-daemon-4nbwg\" (UID: \"966fc2f5-1448-4a14-8b27-21e714628e1a\") " pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:07.881915 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880914    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80c7a3a0-1833-46d5-bd61-7843b21bd530-cni-binary-copy\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.882528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880920    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/460481eb-fc51-49b6-b8f7-7e892d0d0e64-tmp-dir\") pod \"node-resolver-lmgzk\" (UID: \"460481eb-fc51-49b6-b8f7-7e892d0d0e64\") " pod="openshift-dns/node-resolver-lmgzk"
Jul 17 12:22:07.882528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880942    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-cni-netd\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.882528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880955    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-cnibin\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.882528 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:07.880961    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:07.882528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880968    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-multus-socket-dir-parent\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.882528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.880985    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-host-var-lib-cni-bin\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.882528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881005    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/79ba2047-ca88-49e0-8b8e-621b724dd8d0-socket-dir\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.882528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881037    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-cnibin\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.882528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881049    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-cni-netd\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.882528 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:07.881053    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs podName:966fc2f5-1448-4a14-8b27-21e714628e1a nodeName:}" failed. No retries permitted until 2026-07-17 12:22:08.381011637 +0000 UTC m=+3.385058496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs") pod "network-metrics-daemon-4nbwg" (UID: "966fc2f5-1448-4a14-8b27-21e714628e1a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:07.882528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881087    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/79ba2047-ca88-49e0-8b8e-621b724dd8d0-registration-dir\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.882528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881100    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-multus-socket-dir-parent\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.882528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881113    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-etc-openvswitch\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.882528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881120    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/79ba2047-ca88-49e0-8b8e-621b724dd8d0-socket-dir\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.882528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881132    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-host-var-lib-cni-bin\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.882528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881135    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-run-ovn\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.882528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881167    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/79ba2047-ca88-49e0-8b8e-621b724dd8d0-registration-dir\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.883071 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881167    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-cni-bin\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.883071 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881179    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-run-ovn\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.883071 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881193    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-cni-bin\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.883071 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881192    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 12:22:07.883071 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881224    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-etc-openvswitch\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.883071 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881192    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-system-cni-dir\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.883071 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881249    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-os-release\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.883071 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881198    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/460481eb-fc51-49b6-b8f7-7e892d0d0e64-tmp-dir\") pod \"node-resolver-lmgzk\" (UID: \"460481eb-fc51-49b6-b8f7-7e892d0d0e64\") " pod="openshift-dns/node-resolver-lmgzk"
Jul 17 12:22:07.883071 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881264    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.883071 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881304    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-node-log\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.883071 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881328    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-log-socket\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.883071 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881339    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-node-log\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.883071 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881370    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret\") pod \"global-pull-secret-syncer-nw9sw\" (UID: \"a4b940c7-3fe3-4290-b253-efde9655c166\") " pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:07.883071 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881389    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-multus-cni-dir\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.883071 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881410    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-log-socket\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.883071 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:07.881465    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:07.883071 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881465    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-multus-conf-dir\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.883071 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881496    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-multus-conf-dir\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.883536 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881499    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-multus-cni-dir\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.883536 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881496    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-cnibin\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.883536 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:07.881551    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret podName:a4b940c7-3fe3-4290-b253-efde9655c166 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:08.381534963 +0000 UTC m=+3.385581823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret") pod "global-pull-secret-syncer-nw9sw" (UID: "a4b940c7-3fe3-4290-b253-efde9655c166") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:07.883536 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881581    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-systemd\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.883536 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881602    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-kubelet\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.883536 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881621    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-ovn-node-metrics-cert\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.883536 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881651    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-host-kubelet\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.883536 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881673    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a4b940c7-3fe3-4290-b253-efde9655c166-kubelet-config\") pod \"global-pull-secret-syncer-nw9sw\" (UID: \"a4b940c7-3fe3-4290-b253-efde9655c166\") " pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:07.883536 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881689    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2bedbc3-150b-4334-a07f-574108c2901a-host\") pod \"node-ca-q7w4l\" (UID: \"a2bedbc3-150b-4334-a07f-574108c2901a\") " pod="openshift-image-registry/node-ca-q7w4l"
Jul 17 12:22:07.883536 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881704    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-etc-kubernetes\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.883536 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881720    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2rnf\" (UniqueName: \"kubernetes.io/projected/80c7a3a0-1833-46d5-bd61-7843b21bd530-kube-api-access-q2rnf\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.883536 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881746    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mbx6\" (UniqueName: \"kubernetes.io/projected/298d5722-f081-490f-bc9c-dd2cd7bfba23-kube-api-access-9mbx6\") pod \"iptables-alerter-xp78t\" (UID: \"298d5722-f081-490f-bc9c-dd2cd7bfba23\") " pod="openshift-network-operator/iptables-alerter-xp78t"
Jul 17 12:22:07.883536 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881764    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a4b940c7-3fe3-4290-b253-efde9655c166-kubelet-config\") pod \"global-pull-secret-syncer-nw9sw\" (UID: \"a4b940c7-3fe3-4290-b253-efde9655c166\") " pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:07.883536 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881782    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80c7a3a0-1833-46d5-bd61-7843b21bd530-etc-kubernetes\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.883536 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881794    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2bedbc3-150b-4334-a07f-574108c2901a-host\") pod \"node-ca-q7w4l\" (UID: \"a2bedbc3-150b-4334-a07f-574108c2901a\") " pod="openshift-image-registry/node-ca-q7w4l"
Jul 17 12:22:07.883536 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881836    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9916c7a-cfa6-4b17-bea4-0642e4dce23e-konnectivity-ca\") pod \"konnectivity-agent-c2c46\" (UID: \"a9916c7a-cfa6-4b17-bea4-0642e4dce23e\") " pod="kube-system/konnectivity-agent-c2c46"
Jul 17 12:22:07.883536 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.881865    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpp7c\" (UniqueName: \"kubernetes.io/projected/460481eb-fc51-49b6-b8f7-7e892d0d0e64-kube-api-access-gpp7c\") pod \"node-resolver-lmgzk\" (UID: \"460481eb-fc51-49b6-b8f7-7e892d0d0e64\") " pod="openshift-dns/node-resolver-lmgzk"
Jul 17 12:22:07.884018 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.882364    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a9916c7a-cfa6-4b17-bea4-0642e4dce23e-konnectivity-ca\") pod \"konnectivity-agent-c2c46\" (UID: \"a9916c7a-cfa6-4b17-bea4-0642e4dce23e\") " pod="kube-system/konnectivity-agent-c2c46"
Jul 17 12:22:07.884441 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.884421    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-ovn-node-metrics-cert\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.884538 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.884521    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a9916c7a-cfa6-4b17-bea4-0642e4dce23e-agent-certs\") pod \"konnectivity-agent-c2c46\" (UID: \"a9916c7a-cfa6-4b17-bea4-0642e4dce23e\") " pod="kube-system/konnectivity-agent-c2c46"
Jul 17 12:22:07.885298 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:07.885283    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:22:07.885378 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:07.885301    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:22:07.885378 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:07.885313    2569 projected.go:196] Error preparing data for projected volume kube-api-access-fbxdw for pod openshift-network-diagnostics/network-check-target-hcnj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:07.885378 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:07.885366    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw podName:db4fbe66-76bb-480e-9a7f-893b764ec3bf nodeName:}" failed. No retries permitted until 2026-07-17 12:22:08.385350217 +0000 UTC m=+3.389397095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fbxdw" (UniqueName: "kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw") pod "network-check-target-hcnj4" (UID: "db4fbe66-76bb-480e-9a7f-893b764ec3bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:07.886511 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.886483    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lvsm\" (UniqueName: \"kubernetes.io/projected/966fc2f5-1448-4a14-8b27-21e714628e1a-kube-api-access-7lvsm\") pod \"network-metrics-daemon-4nbwg\" (UID: \"966fc2f5-1448-4a14-8b27-21e714628e1a\") " pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:07.887359 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.887343    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn4rf\" (UniqueName: \"kubernetes.io/projected/a2bedbc3-150b-4334-a07f-574108c2901a-kube-api-access-wn4rf\") pod \"node-ca-q7w4l\" (UID: \"a2bedbc3-150b-4334-a07f-574108c2901a\") " pod="openshift-image-registry/node-ca-q7w4l"
Jul 17 12:22:07.889942 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.889922    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqp9z\" (UniqueName: \"kubernetes.io/projected/79ba2047-ca88-49e0-8b8e-621b724dd8d0-kube-api-access-gqp9z\") pod \"aws-ebs-csi-driver-node-zwm8g\" (UID: \"79ba2047-ca88-49e0-8b8e-621b724dd8d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:07.890596 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.890578    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt997\" (UniqueName: \"kubernetes.io/projected/fa1a19dd-c085-43ae-8dc2-d5775ccd5f59-kube-api-access-jt997\") pod \"ovnkube-node-27xxg\" (UID: \"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59\") " pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:07.890698 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.890617    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2rnf\" (UniqueName: \"kubernetes.io/projected/80c7a3a0-1833-46d5-bd61-7843b21bd530-kube-api-access-q2rnf\") pod \"multus-g4c6m\" (UID: \"80c7a3a0-1833-46d5-bd61-7843b21bd530\") " pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:07.891173 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.891158    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpp7c\" (UniqueName: \"kubernetes.io/projected/460481eb-fc51-49b6-b8f7-7e892d0d0e64-kube-api-access-gpp7c\") pod \"node-resolver-lmgzk\" (UID: \"460481eb-fc51-49b6-b8f7-7e892d0d0e64\") " pod="openshift-dns/node-resolver-lmgzk"
Jul 17 12:22:07.983269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983186    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-cnibin\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.983269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983227    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-systemd\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.983269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983249    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mbx6\" (UniqueName: \"kubernetes.io/projected/298d5722-f081-490f-bc9c-dd2cd7bfba23-kube-api-access-9mbx6\") pod \"iptables-alerter-xp78t\" (UID: \"298d5722-f081-490f-bc9c-dd2cd7bfba23\") " pod="openshift-network-operator/iptables-alerter-xp78t"
Jul 17 12:22:07.983509 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983298    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.983509 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983308    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-cnibin\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.983509 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983328    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-kubernetes\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.983509 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983354    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.983509 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983373    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-kubernetes\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.983509 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983376    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-sysctl-d\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.983509 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983327    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-systemd\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.983509 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983398    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-sysctl-conf\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.983509 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983427    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-lib-modules\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.983509 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983441    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-tuned\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.983509 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983467    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-host\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.983509 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983490    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-sys\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.983509 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983511    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-var-lib-kubelet\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.984109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983528    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.984109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983534    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-tmp\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.984109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983573    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm4rd\" (UniqueName: \"kubernetes.io/projected/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-kube-api-access-hm4rd\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.984109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983604    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-modprobe-d\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.984109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983628    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6cgp\" (UniqueName: \"kubernetes.io/projected/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-kube-api-access-m6cgp\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.984109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983635    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-lib-modules\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.984109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983653    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/298d5722-f081-490f-bc9c-dd2cd7bfba23-iptables-alerter-script\") pod \"iptables-alerter-xp78t\" (UID: \"298d5722-f081-490f-bc9c-dd2cd7bfba23\") " pod="openshift-network-operator/iptables-alerter-xp78t"
Jul 17 12:22:07.984109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983677    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/298d5722-f081-490f-bc9c-dd2cd7bfba23-host-slash\") pod \"iptables-alerter-xp78t\" (UID: \"298d5722-f081-490f-bc9c-dd2cd7bfba23\") " pod="openshift-network-operator/iptables-alerter-xp78t"
Jul 17 12:22:07.984109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983702    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.984109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983730    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-sysconfig\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.984109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983763    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-sysctl-d\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.984109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983785    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-run\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.984109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983825    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-system-cni-dir\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.984109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983864    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-os-release\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.984109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983877    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-sysctl-conf\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.984109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983886    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.984109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.983944    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-modprobe-d\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.984879 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.984177    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-host\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.984879 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.984319    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/298d5722-f081-490f-bc9c-dd2cd7bfba23-iptables-alerter-script\") pod \"iptables-alerter-xp78t\" (UID: \"298d5722-f081-490f-bc9c-dd2cd7bfba23\") " pod="openshift-network-operator/iptables-alerter-xp78t"
Jul 17 12:22:07.984879 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.984339    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-sys\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.984879 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.984389    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-var-lib-kubelet\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.984879 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.984413    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-sysconfig\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.984879 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.984427    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/298d5722-f081-490f-bc9c-dd2cd7bfba23-host-slash\") pod \"iptables-alerter-xp78t\" (UID: \"298d5722-f081-490f-bc9c-dd2cd7bfba23\") " pod="openshift-network-operator/iptables-alerter-xp78t"
Jul 17 12:22:07.984879 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.984433    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-system-cni-dir\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.984879 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.984454    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.984879 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.984469    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-run\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.984879 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.984510    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-os-release\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.984879 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.984836    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.985409 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.984912    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.985991 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.985968    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-tmp\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.986290 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.986271    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-etc-tuned\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.992763 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.992723    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6cgp\" (UniqueName: \"kubernetes.io/projected/ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64-kube-api-access-m6cgp\") pod \"tuned-f7cdh\" (UID: \"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64\") " pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:07.993422 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.993397    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mbx6\" (UniqueName: \"kubernetes.io/projected/298d5722-f081-490f-bc9c-dd2cd7bfba23-kube-api-access-9mbx6\") pod \"iptables-alerter-xp78t\" (UID: \"298d5722-f081-490f-bc9c-dd2cd7bfba23\") " pod="openshift-network-operator/iptables-alerter-xp78t"
Jul 17 12:22:07.993575 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.993553    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm4rd\" (UniqueName: \"kubernetes.io/projected/2680a751-e4a6-438e-bcd9-51f8ab07fa3b-kube-api-access-hm4rd\") pod \"multus-additional-cni-plugins-whc8l\" (UID: \"2680a751-e4a6-438e-bcd9-51f8ab07fa3b\") " pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:07.994293 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:07.994277    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g"
Jul 17 12:22:08.000698 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.000676    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c2c46"
Jul 17 12:22:08.007529 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.007477    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q7w4l"
Jul 17 12:22:08.007650 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:08.007626    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ba2047_ca88_49e0_8b8e_621b724dd8d0.slice/crio-2ad83fbf2fc2d98e81598e37427d2aab21973b7ff372f65e9d78798d71ad5688 WatchSource:0}: Error finding container 2ad83fbf2fc2d98e81598e37427d2aab21973b7ff372f65e9d78798d71ad5688: Status 404 returned error can't find the container with id 2ad83fbf2fc2d98e81598e37427d2aab21973b7ff372f65e9d78798d71ad5688
Jul 17 12:22:08.011335 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.011304    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g4c6m"
Jul 17 12:22:08.011621 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:08.011517    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9916c7a_cfa6_4b17_bea4_0642e4dce23e.slice/crio-320b67a07175822a0d01a0fd03054b9c926ef4c46bb074f004f403631b35db94 WatchSource:0}: Error finding container 320b67a07175822a0d01a0fd03054b9c926ef4c46bb074f004f403631b35db94: Status 404 returned error can't find the container with id 320b67a07175822a0d01a0fd03054b9c926ef4c46bb074f004f403631b35db94
Jul 17 12:22:08.015890 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.015850    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lmgzk"
Jul 17 12:22:08.018763 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:08.018712    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2bedbc3_150b_4334_a07f_574108c2901a.slice/crio-b502e696787cd1cde5f6923d8e53d40521a9b1901bd98aee6b5a876947da95b2 WatchSource:0}: Error finding container b502e696787cd1cde5f6923d8e53d40521a9b1901bd98aee6b5a876947da95b2: Status 404 returned error can't find the container with id b502e696787cd1cde5f6923d8e53d40521a9b1901bd98aee6b5a876947da95b2
Jul 17 12:22:08.021903 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.021735    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:08.022509 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:08.022460    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c7a3a0_1833_46d5_bd61_7843b21bd530.slice/crio-5500576f3c87c9c81214485d47e7d36d7c218c0f076d3e901cf2f7643675c002 WatchSource:0}: Error finding container 5500576f3c87c9c81214485d47e7d36d7c218c0f076d3e901cf2f7643675c002: Status 404 returned error can't find the container with id 5500576f3c87c9c81214485d47e7d36d7c218c0f076d3e901cf2f7643675c002
Jul 17 12:22:08.026813 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.026791    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xp78t"
Jul 17 12:22:08.031131 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.030978    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-whc8l"
Jul 17 12:22:08.031230 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:08.031134    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460481eb_fc51_49b6_b8f7_7e892d0d0e64.slice/crio-a5bf99e1584a4f8e8c34825736bb31500ce081bc1c80f1bc49a2c48aeb98fc70 WatchSource:0}: Error finding container a5bf99e1584a4f8e8c34825736bb31500ce081bc1c80f1bc49a2c48aeb98fc70: Status 404 returned error can't find the container with id a5bf99e1584a4f8e8c34825736bb31500ce081bc1c80f1bc49a2c48aeb98fc70
Jul 17 12:22:08.035356 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.035338    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f7cdh"
Jul 17 12:22:08.035856 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:08.035823    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1a19dd_c085_43ae_8dc2_d5775ccd5f59.slice/crio-dccf63d82b79106c04e6a15c664142db8fc856c45dd0f981cdb7c4e513d9e3aa WatchSource:0}: Error finding container dccf63d82b79106c04e6a15c664142db8fc856c45dd0f981cdb7c4e513d9e3aa: Status 404 returned error can't find the container with id dccf63d82b79106c04e6a15c664142db8fc856c45dd0f981cdb7c4e513d9e3aa
Jul 17 12:22:08.037673 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:08.037627    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298d5722_f081_490f_bc9c_dd2cd7bfba23.slice/crio-1e9c88f4a3876fa1a62d84027698d97ffeaab89f44f1bf8c56ac4510d119a0ce WatchSource:0}: Error finding container 1e9c88f4a3876fa1a62d84027698d97ffeaab89f44f1bf8c56ac4510d119a0ce: Status 404 returned error can't find the container with id 1e9c88f4a3876fa1a62d84027698d97ffeaab89f44f1bf8c56ac4510d119a0ce
Jul 17 12:22:08.046070 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:08.046044    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2680a751_e4a6_438e_bcd9_51f8ab07fa3b.slice/crio-4dd0c1cf140c5ce374622725f7e9ff8c253e980e6bbb3087d992471861fd8530 WatchSource:0}: Error finding container 4dd0c1cf140c5ce374622725f7e9ff8c253e980e6bbb3087d992471861fd8530: Status 404 returned error can't find the container with id 4dd0c1cf140c5ce374622725f7e9ff8c253e980e6bbb3087d992471861fd8530
Jul 17 12:22:08.047473 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:08.047447    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5f4cf7_3d44_4f65_af8a_a8e9a6af8a64.slice/crio-b54a7df8e076fb39da0fc510c80a863deb634770d069901d619c54700cfb04f0 WatchSource:0}: Error finding container b54a7df8e076fb39da0fc510c80a863deb634770d069901d619c54700cfb04f0: Status 404 returned error can't find the container with id b54a7df8e076fb39da0fc510c80a863deb634770d069901d619c54700cfb04f0
Jul 17 12:22:08.387000 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.386919    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbxdw\" (UniqueName: \"kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw\") pod \"network-check-target-hcnj4\" (UID: \"db4fbe66-76bb-480e-9a7f-893b764ec3bf\") " pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:08.387000 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.386972    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs\") pod \"network-metrics-daemon-4nbwg\" (UID: \"966fc2f5-1448-4a14-8b27-21e714628e1a\") " pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:08.387203 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.387005    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret\") pod \"global-pull-secret-syncer-nw9sw\" (UID: \"a4b940c7-3fe3-4290-b253-efde9655c166\") " pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:08.387259 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:08.387200    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:08.387311 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:08.387260    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret podName:a4b940c7-3fe3-4290-b253-efde9655c166 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:09.387241985 +0000 UTC m=+4.391288847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret") pod "global-pull-secret-syncer-nw9sw" (UID: "a4b940c7-3fe3-4290-b253-efde9655c166") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:08.388005 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:08.387867    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:22:08.388005 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:08.387891    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:22:08.388005 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:08.387907    2569 projected.go:196] Error preparing data for projected volume kube-api-access-fbxdw for pod openshift-network-diagnostics/network-check-target-hcnj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:08.388005 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:08.387957    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw podName:db4fbe66-76bb-480e-9a7f-893b764ec3bf nodeName:}" failed. No retries permitted until 2026-07-17 12:22:09.387940939 +0000 UTC m=+4.391987861 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbxdw" (UniqueName: "kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw") pod "network-check-target-hcnj4" (UID: "db4fbe66-76bb-480e-9a7f-893b764ec3bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:08.388259 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:08.387987    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:08.388259 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:08.388056    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs podName:966fc2f5-1448-4a14-8b27-21e714628e1a nodeName:}" failed. No retries permitted until 2026-07-17 12:22:09.38804045 +0000 UTC m=+4.392087310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs") pod "network-metrics-daemon-4nbwg" (UID: "966fc2f5-1448-4a14-8b27-21e714628e1a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:08.684992 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.684786    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f7cdh" event={"ID":"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64","Type":"ContainerStarted","Data":"b54a7df8e076fb39da0fc510c80a863deb634770d069901d619c54700cfb04f0"}
Jul 17 12:22:08.689096 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.688999    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whc8l" event={"ID":"2680a751-e4a6-438e-bcd9-51f8ab07fa3b","Type":"ContainerStarted","Data":"4dd0c1cf140c5ce374622725f7e9ff8c253e980e6bbb3087d992471861fd8530"}
Jul 17 12:22:08.695074 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.694945    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg" event={"ID":"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59","Type":"ContainerStarted","Data":"dccf63d82b79106c04e6a15c664142db8fc856c45dd0f981cdb7c4e513d9e3aa"}
Jul 17 12:22:08.698194 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.698107    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lmgzk" event={"ID":"460481eb-fc51-49b6-b8f7-7e892d0d0e64","Type":"ContainerStarted","Data":"a5bf99e1584a4f8e8c34825736bb31500ce081bc1c80f1bc49a2c48aeb98fc70"}
Jul 17 12:22:08.700882 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.700796    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g4c6m" event={"ID":"80c7a3a0-1833-46d5-bd61-7843b21bd530","Type":"ContainerStarted","Data":"5500576f3c87c9c81214485d47e7d36d7c218c0f076d3e901cf2f7643675c002"}
Jul 17 12:22:08.704889 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.704856    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c2c46" event={"ID":"a9916c7a-cfa6-4b17-bea4-0642e4dce23e","Type":"ContainerStarted","Data":"320b67a07175822a0d01a0fd03054b9c926ef4c46bb074f004f403631b35db94"}
Jul 17 12:22:08.708895 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.708865    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xp78t" event={"ID":"298d5722-f081-490f-bc9c-dd2cd7bfba23","Type":"ContainerStarted","Data":"1e9c88f4a3876fa1a62d84027698d97ffeaab89f44f1bf8c56ac4510d119a0ce"}
Jul 17 12:22:08.710482 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.710436    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 12:17:07 +0000 UTC" deadline="2028-01-18 21:11:15.925776983 +0000 UTC"
Jul 17 12:22:08.710482 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.710481    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13208h49m7.215299228s"
Jul 17 12:22:08.711694 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.711653    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q7w4l" event={"ID":"a2bedbc3-150b-4334-a07f-574108c2901a","Type":"ContainerStarted","Data":"b502e696787cd1cde5f6923d8e53d40521a9b1901bd98aee6b5a876947da95b2"}
Jul 17 12:22:08.713161 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:08.713101    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g" event={"ID":"79ba2047-ca88-49e0-8b8e-621b724dd8d0","Type":"ContainerStarted","Data":"2ad83fbf2fc2d98e81598e37427d2aab21973b7ff372f65e9d78798d71ad5688"}
Jul 17 12:22:09.396157 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:09.395189    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbxdw\" (UniqueName: \"kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw\") pod \"network-check-target-hcnj4\" (UID: \"db4fbe66-76bb-480e-9a7f-893b764ec3bf\") " pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:09.396157 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:09.395245    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs\") pod \"network-metrics-daemon-4nbwg\" (UID: \"966fc2f5-1448-4a14-8b27-21e714628e1a\") " pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:09.396157 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:09.395280    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret\") pod \"global-pull-secret-syncer-nw9sw\" (UID: \"a4b940c7-3fe3-4290-b253-efde9655c166\") " pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:09.396157 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:09.395402    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:09.396157 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:09.395461    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret podName:a4b940c7-3fe3-4290-b253-efde9655c166 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:11.395444887 +0000 UTC m=+6.399491750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret") pod "global-pull-secret-syncer-nw9sw" (UID: "a4b940c7-3fe3-4290-b253-efde9655c166") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:09.396157 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:09.395876    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:22:09.396157 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:09.395898    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:22:09.396157 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:09.395910    2569 projected.go:196] Error preparing data for projected volume kube-api-access-fbxdw for pod openshift-network-diagnostics/network-check-target-hcnj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:09.396157 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:09.395988    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw podName:db4fbe66-76bb-480e-9a7f-893b764ec3bf nodeName:}" failed. No retries permitted until 2026-07-17 12:22:11.395973178 +0000 UTC m=+6.400020041 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbxdw" (UniqueName: "kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw") pod "network-check-target-hcnj4" (UID: "db4fbe66-76bb-480e-9a7f-893b764ec3bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:09.396157 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:09.396052    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:09.396157 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:09.396083    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs podName:966fc2f5-1448-4a14-8b27-21e714628e1a nodeName:}" failed. No retries permitted until 2026-07-17 12:22:11.396072532 +0000 UTC m=+6.400119397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs") pod "network-metrics-daemon-4nbwg" (UID: "966fc2f5-1448-4a14-8b27-21e714628e1a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:09.630083 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:09.630050    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:09.630244 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:09.630194    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:22:09.630361 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:09.630317    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:09.630424 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:09.630398    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nw9sw" podUID="a4b940c7-3fe3-4290-b253-efde9655c166"
Jul 17 12:22:09.630479 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:09.630457    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:09.630529 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:09.630516    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hcnj4" podUID="db4fbe66-76bb-480e-9a7f-893b764ec3bf"
Jul 17 12:22:09.711629 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:09.711505    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 12:17:07 +0000 UTC" deadline="2028-02-11 14:21:14.268421745 +0000 UTC"
Jul 17 12:22:09.711629 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:09.711546    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13777h59m4.556880916s"
Jul 17 12:22:11.412971 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:11.412933    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbxdw\" (UniqueName: \"kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw\") pod \"network-check-target-hcnj4\" (UID: \"db4fbe66-76bb-480e-9a7f-893b764ec3bf\") " pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:11.413702 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:11.413001    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs\") pod \"network-metrics-daemon-4nbwg\" (UID: \"966fc2f5-1448-4a14-8b27-21e714628e1a\") " pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:11.413702 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:11.413041    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret\") pod \"global-pull-secret-syncer-nw9sw\" (UID: \"a4b940c7-3fe3-4290-b253-efde9655c166\") " pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:11.413702 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:11.413194    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:11.413702 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:11.413257    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret podName:a4b940c7-3fe3-4290-b253-efde9655c166 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:15.413239197 +0000 UTC m=+10.417286060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret") pod "global-pull-secret-syncer-nw9sw" (UID: "a4b940c7-3fe3-4290-b253-efde9655c166") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:11.413702 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:11.413663    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:22:11.413702 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:11.413682    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:22:11.413702 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:11.413694    2569 projected.go:196] Error preparing data for projected volume kube-api-access-fbxdw for pod openshift-network-diagnostics/network-check-target-hcnj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:11.414120 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:11.413738    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw podName:db4fbe66-76bb-480e-9a7f-893b764ec3bf nodeName:}" failed. No retries permitted until 2026-07-17 12:22:15.41372443 +0000 UTC m=+10.417771298 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbxdw" (UniqueName: "kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw") pod "network-check-target-hcnj4" (UID: "db4fbe66-76bb-480e-9a7f-893b764ec3bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:11.414120 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:11.413815    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:11.414120 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:11.413849    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs podName:966fc2f5-1448-4a14-8b27-21e714628e1a nodeName:}" failed. No retries permitted until 2026-07-17 12:22:15.413836603 +0000 UTC m=+10.417883469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs") pod "network-metrics-daemon-4nbwg" (UID: "966fc2f5-1448-4a14-8b27-21e714628e1a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:11.630925 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:11.630890    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:11.631085 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:11.631016    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nw9sw" podUID="a4b940c7-3fe3-4290-b253-efde9655c166"
Jul 17 12:22:11.631559 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:11.631368    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:11.631559 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:11.631369    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:11.631559 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:11.631465    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hcnj4" podUID="db4fbe66-76bb-480e-9a7f-893b764ec3bf"
Jul 17 12:22:11.631838 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:11.631604    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:22:13.629781 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:13.629691    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:13.630229 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:13.629831    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nw9sw" podUID="a4b940c7-3fe3-4290-b253-efde9655c166"
Jul 17 12:22:13.630229 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:13.630066    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:13.630229 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:13.630150    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hcnj4" podUID="db4fbe66-76bb-480e-9a7f-893b764ec3bf"
Jul 17 12:22:13.630229 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:13.630218    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:13.630441 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:13.630307    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:22:15.444003 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:15.443957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret\") pod \"global-pull-secret-syncer-nw9sw\" (UID: \"a4b940c7-3fe3-4290-b253-efde9655c166\") " pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:15.444482 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:15.444036    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbxdw\" (UniqueName: \"kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw\") pod \"network-check-target-hcnj4\" (UID: \"db4fbe66-76bb-480e-9a7f-893b764ec3bf\") " pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:15.444482 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:15.444066    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs\") pod \"network-metrics-daemon-4nbwg\" (UID: \"966fc2f5-1448-4a14-8b27-21e714628e1a\") " pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:15.444482 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:15.444165    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:15.444482 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:15.444214    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs podName:966fc2f5-1448-4a14-8b27-21e714628e1a nodeName:}" failed. No retries permitted until 2026-07-17 12:22:23.444200542 +0000 UTC m=+18.448247402 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs") pod "network-metrics-daemon-4nbwg" (UID: "966fc2f5-1448-4a14-8b27-21e714628e1a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:15.444482 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:15.444468    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:22:15.444778 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:15.444490    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:22:15.444778 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:15.444502    2569 projected.go:196] Error preparing data for projected volume kube-api-access-fbxdw for pod openshift-network-diagnostics/network-check-target-hcnj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:15.444778 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:15.444552    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw podName:db4fbe66-76bb-480e-9a7f-893b764ec3bf nodeName:}" failed. No retries permitted until 2026-07-17 12:22:23.444537509 +0000 UTC m=+18.448584386 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbxdw" (UniqueName: "kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw") pod "network-check-target-hcnj4" (UID: "db4fbe66-76bb-480e-9a7f-893b764ec3bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:15.444778 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:15.444557    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:15.444778 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:15.444598    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret podName:a4b940c7-3fe3-4290-b253-efde9655c166 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:23.444586485 +0000 UTC m=+18.448633347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret") pod "global-pull-secret-syncer-nw9sw" (UID: "a4b940c7-3fe3-4290-b253-efde9655c166") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:15.629908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:15.629251    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:15.629908 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:15.629361    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hcnj4" podUID="db4fbe66-76bb-480e-9a7f-893b764ec3bf"
Jul 17 12:22:15.629908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:15.629736    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:15.629908 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:15.629857    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:22:15.630408 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:15.630279    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:15.630408 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:15.630375    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nw9sw" podUID="a4b940c7-3fe3-4290-b253-efde9655c166"
Jul 17 12:22:17.628961 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:17.628496    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:17.628961 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:17.628493    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:17.628961 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:17.628633    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:22:17.628961 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:17.628671    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hcnj4" podUID="db4fbe66-76bb-480e-9a7f-893b764ec3bf"
Jul 17 12:22:17.628961 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:17.628724    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:17.628961 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:17.628816    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nw9sw" podUID="a4b940c7-3fe3-4290-b253-efde9655c166"
Jul 17 12:22:19.628574 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:19.628534    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:19.628574 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:19.628564    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:19.629055 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:19.628534    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:19.629055 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:19.628651    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hcnj4" podUID="db4fbe66-76bb-480e-9a7f-893b764ec3bf"
Jul 17 12:22:19.629055 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:19.628795    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nw9sw" podUID="a4b940c7-3fe3-4290-b253-efde9655c166"
Jul 17 12:22:19.629055 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:19.628885    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:22:21.628340 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:21.628302    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:21.628806 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:21.628341    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:21.628806 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:21.628316    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:21.628806 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:21.628430    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nw9sw" podUID="a4b940c7-3fe3-4290-b253-efde9655c166"
Jul 17 12:22:21.628806 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:21.628493    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hcnj4" podUID="db4fbe66-76bb-480e-9a7f-893b764ec3bf"
Jul 17 12:22:21.628806 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:21.628573    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:22:23.506220 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:23.506182    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret\") pod \"global-pull-secret-syncer-nw9sw\" (UID: \"a4b940c7-3fe3-4290-b253-efde9655c166\") " pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:23.506664 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:23.506259    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbxdw\" (UniqueName: \"kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw\") pod \"network-check-target-hcnj4\" (UID: \"db4fbe66-76bb-480e-9a7f-893b764ec3bf\") " pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:23.506664 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:23.506294    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs\") pod \"network-metrics-daemon-4nbwg\" (UID: \"966fc2f5-1448-4a14-8b27-21e714628e1a\") " pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:23.506664 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:23.506333    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:23.506664 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:23.506368    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:23.506664 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:23.506400    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret podName:a4b940c7-3fe3-4290-b253-efde9655c166 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:39.506380888 +0000 UTC m=+34.510427758 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret") pod "global-pull-secret-syncer-nw9sw" (UID: "a4b940c7-3fe3-4290-b253-efde9655c166") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:23.506664 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:23.506405    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:22:23.506664 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:23.506417    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs podName:966fc2f5-1448-4a14-8b27-21e714628e1a nodeName:}" failed. No retries permitted until 2026-07-17 12:22:39.506408015 +0000 UTC m=+34.510454884 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs") pod "network-metrics-daemon-4nbwg" (UID: "966fc2f5-1448-4a14-8b27-21e714628e1a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:23.506664 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:23.506424    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:22:23.506664 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:23.506435    2569 projected.go:196] Error preparing data for projected volume kube-api-access-fbxdw for pod openshift-network-diagnostics/network-check-target-hcnj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:23.506664 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:23.506498    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw podName:db4fbe66-76bb-480e-9a7f-893b764ec3bf nodeName:}" failed. No retries permitted until 2026-07-17 12:22:39.506479435 +0000 UTC m=+34.510526312 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbxdw" (UniqueName: "kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw") pod "network-check-target-hcnj4" (UID: "db4fbe66-76bb-480e-9a7f-893b764ec3bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:23.627999 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:23.627959    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:23.628166 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:23.628089    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hcnj4" podUID="db4fbe66-76bb-480e-9a7f-893b764ec3bf"
Jul 17 12:22:23.628166 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:23.628140    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:23.628267 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:23.628133    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:23.628267 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:23.628231    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nw9sw" podUID="a4b940c7-3fe3-4290-b253-efde9655c166"
Jul 17 12:22:23.628359 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:23.628323    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:22:25.629108 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:25.629072    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:25.629623 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:25.629185    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:25.629623 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:25.629215    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:25.629623 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:25.629180    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nw9sw" podUID="a4b940c7-3fe3-4290-b253-efde9655c166"
Jul 17 12:22:25.629623 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:25.629281    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hcnj4" podUID="db4fbe66-76bb-480e-9a7f-893b764ec3bf"
Jul 17 12:22:25.629623 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:25.629365    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:22:26.751799 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:26.751599    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g4c6m" event={"ID":"80c7a3a0-1833-46d5-bd61-7843b21bd530","Type":"ContainerStarted","Data":"df98603ddc49af9c5350e51b5a19552645851f2db696a099c76bd390dd7cd67f"}
Jul 17 12:22:26.770573 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:26.770525    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-g4c6m" podStartSLOduration=3.212566364 podStartE2EDuration="21.770512764s" totalImagesPullingTime="18.5579464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:26.770127936 +0000 UTC m=+21.774174819" watchObservedRunningTime="2026-07-17 12:22:26.770512764 +0000 UTC m=+21.774559646"
Jul 17 12:22:27.628990 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.628827    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:27.629077 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.628863    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:27.629128 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:27.629076    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:22:27.629216 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:27.629133    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nw9sw" podUID="a4b940c7-3fe3-4290-b253-efde9655c166"
Jul 17 12:22:27.629216 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.628892    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:27.629286 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:27.629263    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hcnj4" podUID="db4fbe66-76bb-480e-9a7f-893b764ec3bf"
Jul 17 12:22:27.755599 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.755574    2569 generic.go:356] "Generic (PLEG): container finished" podID="d412a8e9141d4c1c1a722c1bb3d88144" containerID="b997f3a4c4ce2f043fe5743cd14bb518c2d9599269630c803af74fcb4b7aa105" exitCode=0
Jul 17 12:22:27.756313 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.755661    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal" event={"ID":"d412a8e9141d4c1c1a722c1bb3d88144","Type":"ContainerDied","Data":"b997f3a4c4ce2f043fe5743cd14bb518c2d9599269630c803af74fcb4b7aa105"}
Jul 17 12:22:27.757183 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.757161    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f7cdh" event={"ID":"ac5f4cf7-3d44-4f65-af8a-a8e9a6af8a64","Type":"ContainerStarted","Data":"c4f3befaf47a43958338716820b83b963812a552e903a2e2bc17fb311969c8ed"}
Jul 17 12:22:27.758772 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.758729    2569 generic.go:356] "Generic (PLEG): container finished" podID="2680a751-e4a6-438e-bcd9-51f8ab07fa3b" containerID="a6cc06d5477cd51ffb77f85f2fa7b768dc8b9a939b9e054daeb0d87697a3d975" exitCode=0
Jul 17 12:22:27.758871 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.758781    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whc8l" event={"ID":"2680a751-e4a6-438e-bcd9-51f8ab07fa3b","Type":"ContainerDied","Data":"a6cc06d5477cd51ffb77f85f2fa7b768dc8b9a939b9e054daeb0d87697a3d975"}
Jul 17 12:22:27.763820 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.763802    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:22:27.764178 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.764161    2569 generic.go:356] "Generic (PLEG): container finished" podID="fa1a19dd-c085-43ae-8dc2-d5775ccd5f59" containerID="056a8ccc20de75330a2597efe93e32e892bebc13267d811bd3ef9cbcf3c9779b" exitCode=1
Jul 17 12:22:27.764253 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.764228    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg" event={"ID":"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59","Type":"ContainerStarted","Data":"cf44c975d42481e9dc0a96790a45baa4140c9cbcdacc2e1b7f2be94531a62423"}
Jul 17 12:22:27.764318 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.764302    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg" event={"ID":"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59","Type":"ContainerStarted","Data":"1506dd959252e9fa9fbe6c62ec804498d4a8002e370503b25fb22804683b6906"}
Jul 17 12:22:27.764372 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.764357    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg" event={"ID":"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59","Type":"ContainerStarted","Data":"547994a5968d575effc1f9be78b3e567b176a47257a09e3d70e37adfacf56cab"}
Jul 17 12:22:27.764420 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.764408    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg" event={"ID":"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59","Type":"ContainerDied","Data":"056a8ccc20de75330a2597efe93e32e892bebc13267d811bd3ef9cbcf3c9779b"}
Jul 17 12:22:27.764468 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.764448    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg" event={"ID":"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59","Type":"ContainerStarted","Data":"8e537817e45fe7e57af496594f5da1d76cdf5c494c47638efc20f3fbdb1a9bda"}
Jul 17 12:22:27.765612 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.765589    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lmgzk" event={"ID":"460481eb-fc51-49b6-b8f7-7e892d0d0e64","Type":"ContainerStarted","Data":"93ab8975e2695bfa187c79eb7332a3d2f26a01ad6271729ad269f70c6d1e2c11"}
Jul 17 12:22:27.767852 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.767827    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c2c46" event={"ID":"a9916c7a-cfa6-4b17-bea4-0642e4dce23e","Type":"ContainerStarted","Data":"27dfbc43ff79e6b7d2a76b3ffc6d7066ccc19890c7e0924522029e14cb055c54"}
Jul 17 12:22:27.772588 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.771239    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal" podStartSLOduration=21.771223013 podStartE2EDuration="21.771223013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:26.782794227 +0000 UTC m=+21.786841109" watchObservedRunningTime="2026-07-17 12:22:27.771223013 +0000 UTC m=+22.775269896"
Jul 17 12:22:27.773145 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.773112    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-210.ec2.internal" event={"ID":"bc30aca19fb5f39f59f0d9bb7d486e95","Type":"ContainerStarted","Data":"0ef70074e4ee6601cb4d29e89bcdf151843618e4e271813c68bde0ebf8f9c3ab"}
Jul 17 12:22:27.774501 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.774478    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xp78t" event={"ID":"298d5722-f081-490f-bc9c-dd2cd7bfba23","Type":"ContainerStarted","Data":"ee72a32da359840d1715ba36f01ea2cda8a733bfdccb78f5207c31b74ec76e60"}
Jul 17 12:22:27.776338 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.776318    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q7w4l" event={"ID":"a2bedbc3-150b-4334-a07f-574108c2901a","Type":"ContainerStarted","Data":"389d043c5e1ff7a89b61b2db5fd4c965c107e21cdd19afc4bd571e9502ed4d53"}
Jul 17 12:22:27.777726 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.777707    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g" event={"ID":"79ba2047-ca88-49e0-8b8e-621b724dd8d0","Type":"ContainerStarted","Data":"d78abc79d8b2d2bd95bf15ae9c03ee080e190137abb348069d81f56254d98b50"}
Jul 17 12:22:27.784231 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.784172    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c2c46" podStartSLOduration=8.715015227 podStartE2EDuration="22.78415871s" totalImagesPullingTime="14.069143483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:27.783717708 +0000 UTC m=+22.787764590" watchObservedRunningTime="2026-07-17 12:22:27.78415871 +0000 UTC m=+22.788205592"
Jul 17 12:22:27.797742 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.797705    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lmgzk" podStartSLOduration=4.284997227 podStartE2EDuration="22.797691023s" totalImagesPullingTime="18.512693796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:27.797177465 +0000 UTC m=+22.801224347" watchObservedRunningTime="2026-07-17 12:22:27.797691023 +0000 UTC m=+22.801737904"
Jul 17 12:22:27.833566 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.833308    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f7cdh" podStartSLOduration=4.314848428 podStartE2EDuration="22.833284448s" totalImagesPullingTime="18.51843602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:27.833039875 +0000 UTC m=+22.837086756" watchObservedRunningTime="2026-07-17 12:22:27.833284448 +0000 UTC m=+22.837331331"
Jul 17 12:22:27.847362 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.847322    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q7w4l" podStartSLOduration=8.785394447 podStartE2EDuration="22.847307698s" totalImagesPullingTime="14.061913251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:27.847262792 +0000 UTC m=+22.851309675" watchObservedRunningTime="2026-07-17 12:22:27.847307698 +0000 UTC m=+22.851354577"
Jul 17 12:22:27.872149 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:27.872098    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xp78t" podStartSLOduration=13.489706894 podStartE2EDuration="22.87208516s" totalImagesPullingTime="9.382378266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:27.871069999 +0000 UTC m=+22.875116881" watchObservedRunningTime="2026-07-17 12:22:27.87208516 +0000 UTC m=+22.876132041"
Jul 17 12:22:28.781645 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:28.781608    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal" event={"ID":"d412a8e9141d4c1c1a722c1bb3d88144","Type":"ContainerStarted","Data":"0de3d5a9aa37db81f307ad6cac2ac62729ad96b5a5ee1cbbf4c56df60738a16a"}
Jul 17 12:22:28.784181 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:28.784163    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:22:28.785122 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:28.784654    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg" event={"ID":"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59","Type":"ContainerStarted","Data":"8d4f760eb336cdf89d0393216370a405aec2019c233ddbef7e7f3bde31c4792d"}
Jul 17 12:22:28.796241 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:28.796201    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-210.ec2.internal" podStartSLOduration=22.796184716 podStartE2EDuration="22.796184716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:28.796131065 +0000 UTC m=+23.800177948" watchObservedRunningTime="2026-07-17 12:22:28.796184716 +0000 UTC m=+23.800231600"
Jul 17 12:22:28.954681 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:28.954650    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 12:22:29.628466 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:29.628429    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:29.628466 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:29.628455    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:29.628813 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:29.628562    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nw9sw" podUID="a4b940c7-3fe3-4290-b253-efde9655c166"
Jul 17 12:22:29.628813 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:29.628647    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hcnj4" podUID="db4fbe66-76bb-480e-9a7f-893b764ec3bf"
Jul 17 12:22:29.628813 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:29.628695    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:29.628813 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:29.628799    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:22:29.686806 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:29.686697    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T12:22:28.95467744Z","UUID":"da954a8b-b252-47ce-b7e1-d1f69b67071e","Handler":null,"Name":"","Endpoint":""}
Jul 17 12:22:29.688551 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:29.688519    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 12:22:29.688677 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:29.688563    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 12:22:29.788426 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:29.788394    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g" event={"ID":"79ba2047-ca88-49e0-8b8e-621b724dd8d0","Type":"ContainerStarted","Data":"9aeeaf7f0992754957a8e4a1f2b8b68dfeef436a4436b9ad9d2927a95b3bfb49"}
Jul 17 12:22:30.794391 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:30.794360    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:22:30.794875 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:30.794824    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg" event={"ID":"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59","Type":"ContainerStarted","Data":"1ed3894fd8dbbdf20a873ebe28b9e5bdc4f995b2ae21ce874a3557ed0f062cf1"}
Jul 17 12:22:30.797100 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:30.797079    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g" event={"ID":"79ba2047-ca88-49e0-8b8e-621b724dd8d0","Type":"ContainerStarted","Data":"73987597c1a53e04cba679fe010897b24fd79b771dc5ab7a07b5a19d0d3297c3"}
Jul 17 12:22:30.815331 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:30.815279    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwm8g" podStartSLOduration=4.945139493 podStartE2EDuration="25.815265957s" totalImagesPullingTime="20.870126464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:30.814609147 +0000 UTC m=+25.818656028" watchObservedRunningTime="2026-07-17 12:22:30.815265957 +0000 UTC m=+25.819312839"
Jul 17 12:22:31.171007 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:31.170913    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c2c46"
Jul 17 12:22:31.628470 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:31.628386    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:31.628470 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:31.628431    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:31.628683 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:31.628386    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:31.628683 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:31.628533    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:22:31.628683 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:31.628603    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hcnj4" podUID="db4fbe66-76bb-480e-9a7f-893b764ec3bf"
Jul 17 12:22:31.628882 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:31.628677    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nw9sw" podUID="a4b940c7-3fe3-4290-b253-efde9655c166"
Jul 17 12:22:31.884566 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:31.884471    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c2c46"
Jul 17 12:22:31.885832 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:31.885806    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c2c46"
Jul 17 12:22:32.805570 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:32.805448    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:22:32.806060 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:32.806032    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg" event={"ID":"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59","Type":"ContainerStarted","Data":"3b696694bc00425574f61ec77ae42c1d92a3458293ae278f8a2e706f93e89ab6"}
Jul 17 12:22:32.806473 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:32.806446    2569 scope.go:117] "RemoveContainer" containerID="056a8ccc20de75330a2597efe93e32e892bebc13267d811bd3ef9cbcf3c9779b"
Jul 17 12:22:32.806793 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:32.806776    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c2c46"
Jul 17 12:22:33.627944 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:33.627915    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:33.628654 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:33.627950    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:33.628654 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:33.628017    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nw9sw" podUID="a4b940c7-3fe3-4290-b253-efde9655c166"
Jul 17 12:22:33.628654 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:33.627917    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:33.628654 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:33.628117    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hcnj4" podUID="db4fbe66-76bb-480e-9a7f-893b764ec3bf"
Jul 17 12:22:33.628654 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:33.628232    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:22:33.810439 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:33.810405    2569 generic.go:356] "Generic (PLEG): container finished" podID="2680a751-e4a6-438e-bcd9-51f8ab07fa3b" containerID="c096f3d6cf5b26a05d6dc83b507462d2a4a09de3234e4158a03d2838a62621cf" exitCode=0
Jul 17 12:22:33.810687 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:33.810487    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whc8l" event={"ID":"2680a751-e4a6-438e-bcd9-51f8ab07fa3b","Type":"ContainerDied","Data":"c096f3d6cf5b26a05d6dc83b507462d2a4a09de3234e4158a03d2838a62621cf"}
Jul 17 12:22:33.814444 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:33.814417    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:22:33.814848 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:33.814817    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg" event={"ID":"fa1a19dd-c085-43ae-8dc2-d5775ccd5f59","Type":"ContainerStarted","Data":"10f19a9c7e7d59f3c26df97ea5c1e34b98b11686cdd334ad1b4e6b2198dcf783"}
Jul 17 12:22:33.815120 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:33.815100    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:33.815253 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:33.815173    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:33.815318 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:33.815262    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:33.831144 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:33.831118    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:33.831284 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:33.831222    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:22:33.859976 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:33.859919    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg" podStartSLOduration=9.789327803 podStartE2EDuration="28.859903843s" totalImagesPullingTime="19.07057604s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:33.85791212 +0000 UTC m=+28.861959006" watchObservedRunningTime="2026-07-17 12:22:33.859903843 +0000 UTC m=+28.863950779"
Jul 17 12:22:34.820041 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:34.819787    2569 generic.go:356] "Generic (PLEG): container finished" podID="2680a751-e4a6-438e-bcd9-51f8ab07fa3b" containerID="51a779e4b3c2fe27559a4b59e9629e16373864bb415c629309005ae81bdaa696" exitCode=0
Jul 17 12:22:34.820418 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:34.819859    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whc8l" event={"ID":"2680a751-e4a6-438e-bcd9-51f8ab07fa3b","Type":"ContainerDied","Data":"51a779e4b3c2fe27559a4b59e9629e16373864bb415c629309005ae81bdaa696"}
Jul 17 12:22:34.890466 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:34.890433    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nw9sw"]
Jul 17 12:22:34.890639 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:34.890560    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:34.890707 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:34.890666    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nw9sw" podUID="a4b940c7-3fe3-4290-b253-efde9655c166"
Jul 17 12:22:34.893446 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:34.893408    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hcnj4"]
Jul 17 12:22:34.893578 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:34.893507    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:34.893638 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:34.893599    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hcnj4" podUID="db4fbe66-76bb-480e-9a7f-893b764ec3bf"
Jul 17 12:22:34.894034 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:34.894014    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4nbwg"]
Jul 17 12:22:34.894137 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:34.894113    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:34.894231 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:34.894214    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:22:36.628864 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:36.628821    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:36.628864 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:36.628850    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:36.629339 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:36.628821    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:36.629339 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:36.628945    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nw9sw" podUID="a4b940c7-3fe3-4290-b253-efde9655c166"
Jul 17 12:22:36.629339 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:36.629015    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:22:36.629339 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:36.629058    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hcnj4" podUID="db4fbe66-76bb-480e-9a7f-893b764ec3bf"
Jul 17 12:22:36.826953 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:36.826923    2569 generic.go:356] "Generic (PLEG): container finished" podID="2680a751-e4a6-438e-bcd9-51f8ab07fa3b" containerID="1026d2ebdd20271774aadd20834eb7553d81aae03b9c29ae7a0048baf3182fd2" exitCode=0
Jul 17 12:22:36.827117 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:36.826976    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whc8l" event={"ID":"2680a751-e4a6-438e-bcd9-51f8ab07fa3b","Type":"ContainerDied","Data":"1026d2ebdd20271774aadd20834eb7553d81aae03b9c29ae7a0048baf3182fd2"}
Jul 17 12:22:38.628091 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:38.628058    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:38.628542 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:38.628058    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:38.628542 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:38.628199    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:22:38.628542 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:38.628269    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nw9sw" podUID="a4b940c7-3fe3-4290-b253-efde9655c166"
Jul 17 12:22:38.628542 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:38.628058    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:38.628542 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:38.628405    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hcnj4" podUID="db4fbe66-76bb-480e-9a7f-893b764ec3bf"
Jul 17 12:22:39.527060 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.527021    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret\") pod \"global-pull-secret-syncer-nw9sw\" (UID: \"a4b940c7-3fe3-4290-b253-efde9655c166\") " pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:39.527252 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.527193    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbxdw\" (UniqueName: \"kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw\") pod \"network-check-target-hcnj4\" (UID: \"db4fbe66-76bb-480e-9a7f-893b764ec3bf\") " pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:39.527252 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:39.527199    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:39.527252 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.527217    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs\") pod \"network-metrics-daemon-4nbwg\" (UID: \"966fc2f5-1448-4a14-8b27-21e714628e1a\") " pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:39.527407 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:39.527272    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret podName:a4b940c7-3fe3-4290-b253-efde9655c166 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:11.527251857 +0000 UTC m=+66.531298915 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret") pod "global-pull-secret-syncer-nw9sw" (UID: "a4b940c7-3fe3-4290-b253-efde9655c166") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:39.527407 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:39.527305    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:39.527407 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:39.527347    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:22:39.527407 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:39.527369    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:22:39.527407 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:39.527382    2569 projected.go:196] Error preparing data for projected volume kube-api-access-fbxdw for pod openshift-network-diagnostics/network-check-target-hcnj4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:39.527407 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:39.527401    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs podName:966fc2f5-1448-4a14-8b27-21e714628e1a nodeName:}" failed. No retries permitted until 2026-07-17 12:23:11.527385717 +0000 UTC m=+66.531432601 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs") pod "network-metrics-daemon-4nbwg" (UID: "966fc2f5-1448-4a14-8b27-21e714628e1a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:39.527668 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:39.527431    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw podName:db4fbe66-76bb-480e-9a7f-893b764ec3bf nodeName:}" failed. No retries permitted until 2026-07-17 12:23:11.527412548 +0000 UTC m=+66.531459426 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fbxdw" (UniqueName: "kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw") pod "network-check-target-hcnj4" (UID: "db4fbe66-76bb-480e-9a7f-893b764ec3bf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:39.739221 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.739194    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-210.ec2.internal" event="NodeReady"
Jul 17 12:22:39.739598 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.739319    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 12:22:39.774957 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.774924    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6678b8bcb7-d4r2t"]
Jul 17 12:22:39.776834 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.776808    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.779711 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.779586    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 17 12:22:39.779711 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.779617    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 17 12:22:39.779995 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.779982    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5tvcb\""
Jul 17 12:22:39.780286 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.780271    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 17 12:22:39.785995 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.785971    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 17 12:22:39.786523 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.786502    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mq9p4"]
Jul 17 12:22:39.788440 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.788419    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:22:39.789197 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.789177    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4nsvg"]
Jul 17 12:22:39.790865 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.790848    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4nsvg"
Jul 17 12:22:39.791296 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.791276    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 12:22:39.791448 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.791398    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 12:22:39.791539 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.791482    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gk42k\""
Jul 17 12:22:39.793066 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.793046    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6678b8bcb7-d4r2t"]
Jul 17 12:22:39.793470 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.793457    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 12:22:39.793553 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.793484    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 12:22:39.793869 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.793853    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 12:22:39.793945 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.793897    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-btjhs\""
Jul 17 12:22:39.802849 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.802807    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4nsvg"]
Jul 17 12:22:39.803355 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.803338    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mq9p4"]
Jul 17 12:22:39.830092 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.830068    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.830249 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.830096    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5c7v\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-kube-api-access-r5c7v\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.830249 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.830118    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7ab02ba-f207-4ce7-896c-f1cef9abea99-installation-pull-secrets\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.830249 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.830137    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-bound-sa-token\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.830249 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.830197    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-certificates\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.830249 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.830226    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7ab02ba-f207-4ce7-896c-f1cef9abea99-image-registry-private-configuration\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.830249 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.830249    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7ab02ba-f207-4ce7-896c-f1cef9abea99-trusted-ca\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.830494 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.830276    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7ab02ba-f207-4ce7-896c-f1cef9abea99-ca-trust-extracted\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.931177 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.931135    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-certificates\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.931360 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.931190    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9ff568b-7a93-4a5d-9eac-b04e3b285830-cert\") pod \"ingress-canary-4nsvg\" (UID: \"e9ff568b-7a93-4a5d-9eac-b04e3b285830\") " pod="openshift-ingress-canary/ingress-canary-4nsvg"
Jul 17 12:22:39.931360 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.931231    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7ab02ba-f207-4ce7-896c-f1cef9abea99-image-registry-private-configuration\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.931360 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.931258    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:22:39.931360 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.931291    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7ab02ba-f207-4ce7-896c-f1cef9abea99-trusted-ca\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.931360 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.931315    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7ab02ba-f207-4ce7-896c-f1cef9abea99-ca-trust-extracted\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.931621 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.931428    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.931621 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.931453    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5c7v\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-kube-api-access-r5c7v\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.931621 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.931478    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/668cc1ae-8c83-45b9-a875-b65bd76f80ea-config-volume\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:22:39.931621 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.931502    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzmv4\" (UniqueName: \"kubernetes.io/projected/668cc1ae-8c83-45b9-a875-b65bd76f80ea-kube-api-access-mzmv4\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:22:39.931621 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.931527    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7ab02ba-f207-4ce7-896c-f1cef9abea99-installation-pull-secrets\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.931621 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.931542    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-bound-sa-token\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.931621 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.931557    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/668cc1ae-8c83-45b9-a875-b65bd76f80ea-tmp-dir\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:22:39.931621 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.931572    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr4x7\" (UniqueName: \"kubernetes.io/projected/e9ff568b-7a93-4a5d-9eac-b04e3b285830-kube-api-access-cr4x7\") pod \"ingress-canary-4nsvg\" (UID: \"e9ff568b-7a93-4a5d-9eac-b04e3b285830\") " pod="openshift-ingress-canary/ingress-canary-4nsvg"
Jul 17 12:22:39.931958 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:39.931659    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:22:39.931958 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:39.931671    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6678b8bcb7-d4r2t: secret "image-registry-tls" not found
Jul 17 12:22:39.931958 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:39.931722    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls podName:c7ab02ba-f207-4ce7-896c-f1cef9abea99 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:40.431704635 +0000 UTC m=+35.435751500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls") pod "image-registry-6678b8bcb7-d4r2t" (UID: "c7ab02ba-f207-4ce7-896c-f1cef9abea99") : secret "image-registry-tls" not found
Jul 17 12:22:39.931958 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.931845    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-certificates\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.931958 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.931848    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7ab02ba-f207-4ce7-896c-f1cef9abea99-ca-trust-extracted\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.932328 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.932304    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7ab02ba-f207-4ce7-896c-f1cef9abea99-trusted-ca\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.937068 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.937038    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7ab02ba-f207-4ce7-896c-f1cef9abea99-image-registry-private-configuration\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.937201 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.937038    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7ab02ba-f207-4ce7-896c-f1cef9abea99-installation-pull-secrets\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.941056 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.941028    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-bound-sa-token\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:39.941183 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:39.941098    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5c7v\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-kube-api-access-r5c7v\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:40.033058 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.032727    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/668cc1ae-8c83-45b9-a875-b65bd76f80ea-config-volume\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:22:40.033058 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.033024    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzmv4\" (UniqueName: \"kubernetes.io/projected/668cc1ae-8c83-45b9-a875-b65bd76f80ea-kube-api-access-mzmv4\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:22:40.033278 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.033064    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/668cc1ae-8c83-45b9-a875-b65bd76f80ea-tmp-dir\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:22:40.033278 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.033093    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr4x7\" (UniqueName: \"kubernetes.io/projected/e9ff568b-7a93-4a5d-9eac-b04e3b285830-kube-api-access-cr4x7\") pod \"ingress-canary-4nsvg\" (UID: \"e9ff568b-7a93-4a5d-9eac-b04e3b285830\") " pod="openshift-ingress-canary/ingress-canary-4nsvg"
Jul 17 12:22:40.033278 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.033125    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9ff568b-7a93-4a5d-9eac-b04e3b285830-cert\") pod \"ingress-canary-4nsvg\" (UID: \"e9ff568b-7a93-4a5d-9eac-b04e3b285830\") " pod="openshift-ingress-canary/ingress-canary-4nsvg"
Jul 17 12:22:40.033278 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.033153    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:22:40.033498 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:40.033359    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:22:40.033498 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:40.033433    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls podName:668cc1ae-8c83-45b9-a875-b65bd76f80ea nodeName:}" failed. No retries permitted until 2026-07-17 12:22:40.533411005 +0000 UTC m=+35.537457881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls") pod "dns-default-mq9p4" (UID: "668cc1ae-8c83-45b9-a875-b65bd76f80ea") : secret "dns-default-metrics-tls" not found
Jul 17 12:22:40.033498 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.033451    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/668cc1ae-8c83-45b9-a875-b65bd76f80ea-config-volume\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:22:40.033740 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.033720    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/668cc1ae-8c83-45b9-a875-b65bd76f80ea-tmp-dir\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:22:40.036210 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.036184    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9ff568b-7a93-4a5d-9eac-b04e3b285830-cert\") pod \"ingress-canary-4nsvg\" (UID: \"e9ff568b-7a93-4a5d-9eac-b04e3b285830\") " pod="openshift-ingress-canary/ingress-canary-4nsvg"
Jul 17 12:22:40.043292 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.043263    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzmv4\" (UniqueName: \"kubernetes.io/projected/668cc1ae-8c83-45b9-a875-b65bd76f80ea-kube-api-access-mzmv4\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:22:40.043445 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.043363    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr4x7\" (UniqueName: \"kubernetes.io/projected/e9ff568b-7a93-4a5d-9eac-b04e3b285830-kube-api-access-cr4x7\") pod \"ingress-canary-4nsvg\" (UID: \"e9ff568b-7a93-4a5d-9eac-b04e3b285830\") " pod="openshift-ingress-canary/ingress-canary-4nsvg"
Jul 17 12:22:40.107453 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.107412    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4nsvg"
Jul 17 12:22:40.438531 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.438433    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:40.438706 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:40.438628    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:22:40.438706 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:40.438648    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6678b8bcb7-d4r2t: secret "image-registry-tls" not found
Jul 17 12:22:40.438856 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:40.438719    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls podName:c7ab02ba-f207-4ce7-896c-f1cef9abea99 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:41.438695209 +0000 UTC m=+36.442742104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls") pod "image-registry-6678b8bcb7-d4r2t" (UID: "c7ab02ba-f207-4ce7-896c-f1cef9abea99") : secret "image-registry-tls" not found
Jul 17 12:22:40.539729 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.539686    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:22:40.539924 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:40.539881    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:22:40.539983 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:40.539961    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls podName:668cc1ae-8c83-45b9-a875-b65bd76f80ea nodeName:}" failed. No retries permitted until 2026-07-17 12:22:41.53994056 +0000 UTC m=+36.543987439 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls") pod "dns-default-mq9p4" (UID: "668cc1ae-8c83-45b9-a875-b65bd76f80ea") : secret "dns-default-metrics-tls" not found
Jul 17 12:22:40.628115 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.628073    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:22:40.628115 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.628110    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:22:40.628303 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.628110    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:22:40.631531 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.631401    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 12:22:40.631531 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.631454    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 12:22:40.631531 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.631407    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2chv5\""
Jul 17 12:22:40.631531 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.631495    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 12:22:40.631791 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.631536    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vpz7h\""
Jul 17 12:22:40.631791 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:40.631407    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 12:22:41.447302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:41.447272    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:41.447721 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:41.447406    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:22:41.447721 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:41.447419    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6678b8bcb7-d4r2t: secret "image-registry-tls" not found
Jul 17 12:22:41.447721 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:41.447473    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls podName:c7ab02ba-f207-4ce7-896c-f1cef9abea99 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:43.447455578 +0000 UTC m=+38.451502441 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls") pod "image-registry-6678b8bcb7-d4r2t" (UID: "c7ab02ba-f207-4ce7-896c-f1cef9abea99") : secret "image-registry-tls" not found
Jul 17 12:22:41.516107 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:41.516065    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4nsvg"]
Jul 17 12:22:41.547693 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:41.547653    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:22:41.547851 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:41.547816    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:22:41.547891 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:41.547883    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls podName:668cc1ae-8c83-45b9-a875-b65bd76f80ea nodeName:}" failed. No retries permitted until 2026-07-17 12:22:43.547867397 +0000 UTC m=+38.551914257 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls") pod "dns-default-mq9p4" (UID: "668cc1ae-8c83-45b9-a875-b65bd76f80ea") : secret "dns-default-metrics-tls" not found
Jul 17 12:22:41.573343 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:22:41.573304    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ff568b_7a93_4a5d_9eac_b04e3b285830.slice/crio-b7431e85fcbb7ec3f31c81b903c456ec587e8edff9145e9b2a4a2f0672357f1c WatchSource:0}: Error finding container b7431e85fcbb7ec3f31c81b903c456ec587e8edff9145e9b2a4a2f0672357f1c: Status 404 returned error can't find the container with id b7431e85fcbb7ec3f31c81b903c456ec587e8edff9145e9b2a4a2f0672357f1c
Jul 17 12:22:41.841050 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:41.841004    2569 generic.go:356] "Generic (PLEG): container finished" podID="2680a751-e4a6-438e-bcd9-51f8ab07fa3b" containerID="e798289616bb854bf0126be226492fc81f760f43f28e5baac7dca6d773d88e8a" exitCode=0
Jul 17 12:22:41.841216 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:41.841087    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whc8l" event={"ID":"2680a751-e4a6-438e-bcd9-51f8ab07fa3b","Type":"ContainerDied","Data":"e798289616bb854bf0126be226492fc81f760f43f28e5baac7dca6d773d88e8a"}
Jul 17 12:22:41.842112 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:41.842077    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4nsvg" event={"ID":"e9ff568b-7a93-4a5d-9eac-b04e3b285830","Type":"ContainerStarted","Data":"b7431e85fcbb7ec3f31c81b903c456ec587e8edff9145e9b2a4a2f0672357f1c"}
Jul 17 12:22:42.848730 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:42.848694    2569 generic.go:356] "Generic (PLEG): container finished" podID="2680a751-e4a6-438e-bcd9-51f8ab07fa3b" containerID="d282b593c4678176833735c87b201d0df20a496b5d1dbb737533319c5517bf3c" exitCode=0
Jul 17 12:22:42.849532 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:42.848815    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whc8l" event={"ID":"2680a751-e4a6-438e-bcd9-51f8ab07fa3b","Type":"ContainerDied","Data":"d282b593c4678176833735c87b201d0df20a496b5d1dbb737533319c5517bf3c"}
Jul 17 12:22:43.464385 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:43.464130    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:43.464532 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:43.464283    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:22:43.464532 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:43.464422    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6678b8bcb7-d4r2t: secret "image-registry-tls" not found
Jul 17 12:22:43.464532 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:43.464489    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls podName:c7ab02ba-f207-4ce7-896c-f1cef9abea99 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:47.464469019 +0000 UTC m=+42.468515896 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls") pod "image-registry-6678b8bcb7-d4r2t" (UID: "c7ab02ba-f207-4ce7-896c-f1cef9abea99") : secret "image-registry-tls" not found
Jul 17 12:22:43.565028 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:43.564989    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:22:43.565185 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:43.565138    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:22:43.565256 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:43.565198    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls podName:668cc1ae-8c83-45b9-a875-b65bd76f80ea nodeName:}" failed. No retries permitted until 2026-07-17 12:22:47.565182212 +0000 UTC m=+42.569229090 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls") pod "dns-default-mq9p4" (UID: "668cc1ae-8c83-45b9-a875-b65bd76f80ea") : secret "dns-default-metrics-tls" not found
Jul 17 12:22:43.855490 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:43.855455    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whc8l" event={"ID":"2680a751-e4a6-438e-bcd9-51f8ab07fa3b","Type":"ContainerStarted","Data":"ba7f0f103f69ef632be567747870289c9ebe2c17a3dbb378926ec0d091438ff7"}
Jul 17 12:22:43.879735 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:43.879684    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-whc8l" podStartSLOduration=8.616023398 podStartE2EDuration="38.879665788s" totalImagesPullingTime="30.26364239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:43.879257686 +0000 UTC m=+38.883304582" watchObservedRunningTime="2026-07-17 12:22:43.879665788 +0000 UTC m=+38.883712652"
Jul 17 12:22:44.859299 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:44.859267    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4nsvg" event={"ID":"e9ff568b-7a93-4a5d-9eac-b04e3b285830","Type":"ContainerStarted","Data":"a4133e75a875c4c5bf424caf9dc66237f4943a85c3eeeefc2dd1a3a91ad8b183"}
Jul 17 12:22:44.875706 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:44.875658    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4nsvg" podStartSLOduration=3.2158938 podStartE2EDuration="5.875643418s" totalImagesPullingTime="2.659749618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:22:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:44.874020859 +0000 UTC m=+39.878067752" watchObservedRunningTime="2026-07-17 12:22:44.875643418 +0000 UTC m=+39.879690299"
Jul 17 12:22:47.496628 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:47.496589    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:47.497042 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:47.496692    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:22:47.497042 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:47.496703    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6678b8bcb7-d4r2t: secret "image-registry-tls" not found
Jul 17 12:22:47.497042 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:47.496767    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls podName:c7ab02ba-f207-4ce7-896c-f1cef9abea99 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:55.496735926 +0000 UTC m=+50.500782786 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls") pod "image-registry-6678b8bcb7-d4r2t" (UID: "c7ab02ba-f207-4ce7-896c-f1cef9abea99") : secret "image-registry-tls" not found
Jul 17 12:22:47.597645 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:47.597613    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:22:47.597784 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:47.597762    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:22:47.597822 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:47.597818    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls podName:668cc1ae-8c83-45b9-a875-b65bd76f80ea nodeName:}" failed. No retries permitted until 2026-07-17 12:22:55.597803077 +0000 UTC m=+50.601849937 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls") pod "dns-default-mq9p4" (UID: "668cc1ae-8c83-45b9-a875-b65bd76f80ea") : secret "dns-default-metrics-tls" not found
Jul 17 12:22:54.962467 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:54.962435    2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c2c46_a9916c7a-cfa6-4b17-bea4-0642e4dce23e/konnectivity-agent/0.log" limit=1024
Jul 17 12:22:55.557599 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:55.557565    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:22:55.557772 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:55.557665    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:22:55.557772 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:55.557675    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6678b8bcb7-d4r2t: secret "image-registry-tls" not found
Jul 17 12:22:55.557772 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:55.557724    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls podName:c7ab02ba-f207-4ce7-896c-f1cef9abea99 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:11.557709546 +0000 UTC m=+66.561756406 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls") pod "image-registry-6678b8bcb7-d4r2t" (UID: "c7ab02ba-f207-4ce7-896c-f1cef9abea99") : secret "image-registry-tls" not found
Jul 17 12:22:55.658383 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:22:55.658352    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:22:55.658528 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:55.658459    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:22:55.658528 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:22:55.658503    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls podName:668cc1ae-8c83-45b9-a875-b65bd76f80ea nodeName:}" failed. No retries permitted until 2026-07-17 12:23:11.658492014 +0000 UTC m=+66.662538873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls") pod "dns-default-mq9p4" (UID: "668cc1ae-8c83-45b9-a875-b65bd76f80ea") : secret "dns-default-metrics-tls" not found
Jul 17 12:23:05.832724 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:05.832695    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-27xxg"
Jul 17 12:23:11.568036 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:11.567994    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs\") pod \"network-metrics-daemon-4nbwg\" (UID: \"966fc2f5-1448-4a14-8b27-21e714628e1a\") " pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:23:11.568573 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:11.568044    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret\") pod \"global-pull-secret-syncer-nw9sw\" (UID: \"a4b940c7-3fe3-4290-b253-efde9655c166\") " pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:23:11.568573 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:11.568081    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:23:11.568573 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:11.568130    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbxdw\" (UniqueName: \"kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw\") pod \"network-check-target-hcnj4\" (UID: \"db4fbe66-76bb-480e-9a7f-893b764ec3bf\") " pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:23:11.568573 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:23:11.568233    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:23:11.568573 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:23:11.568254    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6678b8bcb7-d4r2t: secret "image-registry-tls" not found
Jul 17 12:23:11.568573 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:23:11.568313    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls podName:c7ab02ba-f207-4ce7-896c-f1cef9abea99 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:43.568297561 +0000 UTC m=+98.572344421 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls") pod "image-registry-6678b8bcb7-d4r2t" (UID: "c7ab02ba-f207-4ce7-896c-f1cef9abea99") : secret "image-registry-tls" not found
Jul 17 12:23:11.577461 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:11.577441    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 12:23:11.577535 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:11.577475    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 12:23:11.577588 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:11.577532    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 12:23:11.578511 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:23:11.578496    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 17 12:23:11.578609 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:23:11.578596    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs podName:966fc2f5-1448-4a14-8b27-21e714628e1a nodeName:}" failed. No retries permitted until 2026-07-17 12:24:15.57857927 +0000 UTC m=+130.582626144 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs") pod "network-metrics-daemon-4nbwg" (UID: "966fc2f5-1448-4a14-8b27-21e714628e1a") : secret "metrics-daemon-secret" not found
Jul 17 12:23:11.580469 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:11.580456    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 12:23:11.582140 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:11.582123    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4b940c7-3fe3-4290-b253-efde9655c166-original-pull-secret\") pod \"global-pull-secret-syncer-nw9sw\" (UID: \"a4b940c7-3fe3-4290-b253-efde9655c166\") " pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:23:11.591184 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:11.591165    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbxdw\" (UniqueName: \"kubernetes.io/projected/db4fbe66-76bb-480e-9a7f-893b764ec3bf-kube-api-access-fbxdw\") pod \"network-check-target-hcnj4\" (UID: \"db4fbe66-76bb-480e-9a7f-893b764ec3bf\") " pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:23:11.669102 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:11.669078    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:23:11.669218 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:23:11.669204    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:23:11.669261 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:23:11.669254    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls podName:668cc1ae-8c83-45b9-a875-b65bd76f80ea nodeName:}" failed. No retries permitted until 2026-07-17 12:23:43.669239983 +0000 UTC m=+98.673286844 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls") pod "dns-default-mq9p4" (UID: "668cc1ae-8c83-45b9-a875-b65bd76f80ea") : secret "dns-default-metrics-tls" not found
Jul 17 12:23:11.839694 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:11.839642    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nw9sw"
Jul 17 12:23:11.849545 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:11.849527    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2chv5\""
Jul 17 12:23:11.857605 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:11.857591    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:23:12.013141 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:12.013112    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nw9sw"]
Jul 17 12:23:12.016736 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:23:12.016704    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b940c7_3fe3_4290_b253_efde9655c166.slice/crio-2703f586130964eeb322ee1b700b2ececd59004509adfca610e082bfa00cbb17 WatchSource:0}: Error finding container 2703f586130964eeb322ee1b700b2ececd59004509adfca610e082bfa00cbb17: Status 404 returned error can't find the container with id 2703f586130964eeb322ee1b700b2ececd59004509adfca610e082bfa00cbb17
Jul 17 12:23:12.040510 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:12.040476    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hcnj4"]
Jul 17 12:23:12.043641 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:23:12.043616    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4fbe66_76bb_480e_9a7f_893b764ec3bf.slice/crio-f61805bb042bff2815de1eb07b5fb41c476616d2c6755af374ac62ca1a80e634 WatchSource:0}: Error finding container f61805bb042bff2815de1eb07b5fb41c476616d2c6755af374ac62ca1a80e634: Status 404 returned error can't find the container with id f61805bb042bff2815de1eb07b5fb41c476616d2c6755af374ac62ca1a80e634
Jul 17 12:23:12.926119 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:12.926056    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hcnj4" event={"ID":"db4fbe66-76bb-480e-9a7f-893b764ec3bf","Type":"ContainerStarted","Data":"f61805bb042bff2815de1eb07b5fb41c476616d2c6755af374ac62ca1a80e634"}
Jul 17 12:23:12.927410 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:12.927378    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nw9sw" event={"ID":"a4b940c7-3fe3-4290-b253-efde9655c166","Type":"ContainerStarted","Data":"2703f586130964eeb322ee1b700b2ececd59004509adfca610e082bfa00cbb17"}
Jul 17 12:23:16.939480 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:16.939441    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nw9sw" event={"ID":"a4b940c7-3fe3-4290-b253-efde9655c166","Type":"ContainerStarted","Data":"f8ebc302c22e22dc6ff197faf1265e5edc9bae1df655d5484597dec5ce061a4b"}
Jul 17 12:23:16.940790 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:16.940770    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hcnj4" event={"ID":"db4fbe66-76bb-480e-9a7f-893b764ec3bf","Type":"ContainerStarted","Data":"726f7fa8e040fc9c80a37821c22bba816031b36de8c3f45ea8db07a8d0e5631a"}
Jul 17 12:23:16.940912 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:16.940900    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:23:16.954284 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:16.954244    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nw9sw" podStartSLOduration=67.519154074 podStartE2EDuration="1m11.954232534s" totalImagesPullingTime="4.43507846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:23:16.953375483 +0000 UTC m=+71.957422377" watchObservedRunningTime="2026-07-17 12:23:16.954232534 +0000 UTC m=+71.958279416"
Jul 17 12:23:16.966924 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:16.966885    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hcnj4" podStartSLOduration=67.562374105 podStartE2EDuration="1m11.966872246s" totalImagesPullingTime="4.404498141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:23:16.966429436 +0000 UTC m=+71.970476321" watchObservedRunningTime="2026-07-17 12:23:16.966872246 +0000 UTC m=+71.970919127"
Jul 17 12:23:43.591371 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:43.591314    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:23:43.591883 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:23:43.591479    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:23:43.591883 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:23:43.591500    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6678b8bcb7-d4r2t: secret "image-registry-tls" not found
Jul 17 12:23:43.591883 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:23:43.591567    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls podName:c7ab02ba-f207-4ce7-896c-f1cef9abea99 nodeName:}" failed. No retries permitted until 2026-07-17 12:24:47.591551206 +0000 UTC m=+162.595598066 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls") pod "image-registry-6678b8bcb7-d4r2t" (UID: "c7ab02ba-f207-4ce7-896c-f1cef9abea99") : secret "image-registry-tls" not found
Jul 17 12:23:43.692324 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:43.692282    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:23:43.692488 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:23:43.692409    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:23:43.692488 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:23:43.692469    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls podName:668cc1ae-8c83-45b9-a875-b65bd76f80ea nodeName:}" failed. No retries permitted until 2026-07-17 12:24:47.692455908 +0000 UTC m=+162.696502769 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls") pod "dns-default-mq9p4" (UID: "668cc1ae-8c83-45b9-a875-b65bd76f80ea") : secret "dns-default-metrics-tls" not found
Jul 17 12:23:47.945972 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:23:47.945945    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hcnj4"
Jul 17 12:24:15.606538 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:24:15.606503    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs\") pod \"network-metrics-daemon-4nbwg\" (UID: \"966fc2f5-1448-4a14-8b27-21e714628e1a\") " pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:24:15.607167 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:24:15.606648    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 17 12:24:15.607167 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:24:15.606732    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs podName:966fc2f5-1448-4a14-8b27-21e714628e1a nodeName:}" failed. No retries permitted until 2026-07-17 12:26:17.606706412 +0000 UTC m=+252.610753291 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs") pod "network-metrics-daemon-4nbwg" (UID: "966fc2f5-1448-4a14-8b27-21e714628e1a") : secret "metrics-daemon-secret" not found
Jul 17 12:24:42.788269 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:24:42.788220    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t" podUID="c7ab02ba-f207-4ce7-896c-f1cef9abea99"
Jul 17 12:24:42.801046 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:24:42.800954    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mq9p4" podUID="668cc1ae-8c83-45b9-a875-b65bd76f80ea"
Jul 17 12:24:43.142668 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:24:43.142592    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:24:43.142829 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:24:43.142597    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:24:43.652711 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:24:43.652671    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:24:47.613999 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:24:47.613963    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls\") pod \"image-registry-6678b8bcb7-d4r2t\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") " pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:24:47.614414 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:24:47.614074    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:24:47.614414 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:24:47.614091    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6678b8bcb7-d4r2t: secret "image-registry-tls" not found
Jul 17 12:24:47.614414 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:24:47.614149    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls podName:c7ab02ba-f207-4ce7-896c-f1cef9abea99 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:49.614131666 +0000 UTC m=+284.618178526 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls") pod "image-registry-6678b8bcb7-d4r2t" (UID: "c7ab02ba-f207-4ce7-896c-f1cef9abea99") : secret "image-registry-tls" not found
Jul 17 12:24:47.714665 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:24:47.714630    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:24:47.714831 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:24:47.714811    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:24:47.714893 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:24:47.714879    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls podName:668cc1ae-8c83-45b9-a875-b65bd76f80ea nodeName:}" failed. No retries permitted until 2026-07-17 12:26:49.714859879 +0000 UTC m=+284.718906755 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls") pod "dns-default-mq9p4" (UID: "668cc1ae-8c83-45b9-a875-b65bd76f80ea") : secret "dns-default-metrics-tls" not found
Jul 17 12:24:57.628617 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:24:57.628515    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:26:17.682144 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:17.682103    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs\") pod \"network-metrics-daemon-4nbwg\" (UID: \"966fc2f5-1448-4a14-8b27-21e714628e1a\") " pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:26:17.682656 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:17.682264    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 17 12:26:17.682656 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:17.682336    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs podName:966fc2f5-1448-4a14-8b27-21e714628e1a nodeName:}" failed. No retries permitted until 2026-07-17 12:28:19.682318435 +0000 UTC m=+374.686365295 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs") pod "network-metrics-daemon-4nbwg" (UID: "966fc2f5-1448-4a14-8b27-21e714628e1a") : secret "metrics-daemon-secret" not found
Jul 17 12:26:27.061936 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:27.061898    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8"]
Jul 17 12:26:27.063726 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:27.063710    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8"
Jul 17 12:26:27.066340 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:27.066316    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wnwlh\""
Jul 17 12:26:27.066476 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:27.066366    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 17 12:26:27.066576 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:27.066562    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 17 12:26:27.067514 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:27.067495    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 17 12:26:27.080162 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:27.080142    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8"]
Jul 17 12:26:27.144383 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:27.144355    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e974b1cb-4c06-48c1-aded-22b183fc908b-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wcgp8\" (UID: \"e974b1cb-4c06-48c1-aded-22b183fc908b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8"
Jul 17 12:26:27.144475 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:27.144388    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9whqg\" (UniqueName: \"kubernetes.io/projected/e974b1cb-4c06-48c1-aded-22b183fc908b-kube-api-access-9whqg\") pod \"cluster-samples-operator-79b7c55f87-wcgp8\" (UID: \"e974b1cb-4c06-48c1-aded-22b183fc908b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8"
Jul 17 12:26:27.245393 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:27.245361    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e974b1cb-4c06-48c1-aded-22b183fc908b-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wcgp8\" (UID: \"e974b1cb-4c06-48c1-aded-22b183fc908b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8"
Jul 17 12:26:27.245492 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:27.245396    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9whqg\" (UniqueName: \"kubernetes.io/projected/e974b1cb-4c06-48c1-aded-22b183fc908b-kube-api-access-9whqg\") pod \"cluster-samples-operator-79b7c55f87-wcgp8\" (UID: \"e974b1cb-4c06-48c1-aded-22b183fc908b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8"
Jul 17 12:26:27.245534 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:27.245512    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 12:26:27.245588 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:27.245576    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e974b1cb-4c06-48c1-aded-22b183fc908b-samples-operator-tls podName:e974b1cb-4c06-48c1-aded-22b183fc908b nodeName:}" failed. No retries permitted until 2026-07-17 12:26:27.745557255 +0000 UTC m=+262.749604166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e974b1cb-4c06-48c1-aded-22b183fc908b-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-wcgp8" (UID: "e974b1cb-4c06-48c1-aded-22b183fc908b") : secret "samples-operator-tls" not found
Jul 17 12:26:27.254027 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:27.253996    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9whqg\" (UniqueName: \"kubernetes.io/projected/e974b1cb-4c06-48c1-aded-22b183fc908b-kube-api-access-9whqg\") pod \"cluster-samples-operator-79b7c55f87-wcgp8\" (UID: \"e974b1cb-4c06-48c1-aded-22b183fc908b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8"
Jul 17 12:26:27.750392 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:27.750361    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e974b1cb-4c06-48c1-aded-22b183fc908b-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wcgp8\" (UID: \"e974b1cb-4c06-48c1-aded-22b183fc908b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8"
Jul 17 12:26:27.750511 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:27.750465    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 12:26:27.750552 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:27.750525    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e974b1cb-4c06-48c1-aded-22b183fc908b-samples-operator-tls podName:e974b1cb-4c06-48c1-aded-22b183fc908b nodeName:}" failed. No retries permitted until 2026-07-17 12:26:28.750510536 +0000 UTC m=+263.754557396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e974b1cb-4c06-48c1-aded-22b183fc908b-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-wcgp8" (UID: "e974b1cb-4c06-48c1-aded-22b183fc908b") : secret "samples-operator-tls" not found
Jul 17 12:26:28.757332 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:28.757280    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e974b1cb-4c06-48c1-aded-22b183fc908b-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wcgp8\" (UID: \"e974b1cb-4c06-48c1-aded-22b183fc908b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8"
Jul 17 12:26:28.757905 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:28.757463    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 12:26:28.757905 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:28.757562    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e974b1cb-4c06-48c1-aded-22b183fc908b-samples-operator-tls podName:e974b1cb-4c06-48c1-aded-22b183fc908b nodeName:}" failed. No retries permitted until 2026-07-17 12:26:30.757536242 +0000 UTC m=+265.761583104 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e974b1cb-4c06-48c1-aded-22b183fc908b-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-wcgp8" (UID: "e974b1cb-4c06-48c1-aded-22b183fc908b") : secret "samples-operator-tls" not found
Jul 17 12:26:30.768949 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.768912    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ctcgp"]
Jul 17 12:26:30.771212 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.771187    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:30.772934 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.772898    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e974b1cb-4c06-48c1-aded-22b183fc908b-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wcgp8\" (UID: \"e974b1cb-4c06-48c1-aded-22b183fc908b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8"
Jul 17 12:26:30.773112 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:30.773087    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 12:26:30.773197 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:30.773184    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e974b1cb-4c06-48c1-aded-22b183fc908b-samples-operator-tls podName:e974b1cb-4c06-48c1-aded-22b183fc908b nodeName:}" failed. No retries permitted until 2026-07-17 12:26:34.773163519 +0000 UTC m=+269.777210379 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e974b1cb-4c06-48c1-aded-22b183fc908b-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-wcgp8" (UID: "e974b1cb-4c06-48c1-aded-22b183fc908b") : secret "samples-operator-tls" not found
Jul 17 12:26:30.776191 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.776169    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 12:26:30.776319 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.776273    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 12:26:30.776680 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.776663    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 12:26:30.777515 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.777499    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 12:26:30.777606 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.777519    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nmxgd\""
Jul 17 12:26:30.797919 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.797887    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ctcgp"]
Jul 17 12:26:30.873879 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.873852    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-data-volume\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:30.873978 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.873883    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:30.873978 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.873915    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhpkf\" (UniqueName: \"kubernetes.io/projected/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-api-access-lhpkf\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:30.873978 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.873966    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-crio-socket\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:30.874088 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.873987    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:30.974341 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.974302    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-data-volume\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:30.974341 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.974343    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:30.974487 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.974370    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhpkf\" (UniqueName: \"kubernetes.io/projected/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-api-access-lhpkf\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:30.974487 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.974390    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-crio-socket\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:30.974487 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.974447    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-crio-socket\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:30.974487 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:30.974473    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:30.974610 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.974504    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:30.974610 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:30.974536    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-insights-runtime-extractor-tls podName:70772ec3-2b0c-4134-9be0-039e0a2c0dc4 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:31.474517517 +0000 UTC m=+266.478564383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ctcgp" (UID: "70772ec3-2b0c-4134-9be0-039e0a2c0dc4") : secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:30.974610 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:30.974600    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 12:26:30.974717 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:30.974686    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-rbac-proxy-cm podName:70772ec3-2b0c-4134-9be0-039e0a2c0dc4 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:31.474667831 +0000 UTC m=+266.478714696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ctcgp" (UID: "70772ec3-2b0c-4134-9be0-039e0a2c0dc4") : configmap "kube-rbac-proxy" not found
Jul 17 12:26:30.974780 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.974724    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-data-volume\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:30.982876 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:30.982854    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhpkf\" (UniqueName: \"kubernetes.io/projected/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-api-access-lhpkf\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:31.478394 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:31.478355    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:31.478525 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:31.478460    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:31.478525 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:31.478478    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 12:26:31.478598 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:31.478576    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-rbac-proxy-cm podName:70772ec3-2b0c-4134-9be0-039e0a2c0dc4 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:32.478558649 +0000 UTC m=+267.482605516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ctcgp" (UID: "70772ec3-2b0c-4134-9be0-039e0a2c0dc4") : configmap "kube-rbac-proxy" not found
Jul 17 12:26:31.478598 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:31.478580    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:31.478673 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:31.478619    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-insights-runtime-extractor-tls podName:70772ec3-2b0c-4134-9be0-039e0a2c0dc4 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:32.478609335 +0000 UTC m=+267.482656231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ctcgp" (UID: "70772ec3-2b0c-4134-9be0-039e0a2c0dc4") : secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:31.558781 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:31.558733    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2g2f7"]
Jul 17 12:26:31.560795 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:31.560778    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2g2f7"
Jul 17 12:26:31.563500 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:31.563477    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 17 12:26:31.563616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:31.563540    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 17 12:26:31.564558 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:31.564539    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-s6hbq\""
Jul 17 12:26:31.568721 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:31.568687    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2g2f7"]
Jul 17 12:26:31.680699 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:31.680669    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnbrf\" (UniqueName: \"kubernetes.io/projected/4bbbe48c-ae26-4550-af9b-77f6efdbc830-kube-api-access-mnbrf\") pod \"migrator-7cc9c7ffd7-2g2f7\" (UID: \"4bbbe48c-ae26-4550-af9b-77f6efdbc830\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2g2f7"
Jul 17 12:26:31.781726 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:31.781662    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnbrf\" (UniqueName: \"kubernetes.io/projected/4bbbe48c-ae26-4550-af9b-77f6efdbc830-kube-api-access-mnbrf\") pod \"migrator-7cc9c7ffd7-2g2f7\" (UID: \"4bbbe48c-ae26-4550-af9b-77f6efdbc830\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2g2f7"
Jul 17 12:26:31.789173 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:31.789147    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnbrf\" (UniqueName: \"kubernetes.io/projected/4bbbe48c-ae26-4550-af9b-77f6efdbc830-kube-api-access-mnbrf\") pod \"migrator-7cc9c7ffd7-2g2f7\" (UID: \"4bbbe48c-ae26-4550-af9b-77f6efdbc830\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2g2f7"
Jul 17 12:26:31.870484 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:31.870461    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2g2f7"
Jul 17 12:26:31.992126 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:31.992098    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2g2f7"]
Jul 17 12:26:31.994352 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:26:31.994327    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bbbe48c_ae26_4550_af9b_77f6efdbc830.slice/crio-afa590d61875fa23f12f89629ccf6ac054d6282f46431ae8135e54ffccca2724 WatchSource:0}: Error finding container afa590d61875fa23f12f89629ccf6ac054d6282f46431ae8135e54ffccca2724: Status 404 returned error can't find the container with id afa590d61875fa23f12f89629ccf6ac054d6282f46431ae8135e54ffccca2724
Jul 17 12:26:32.415528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:32.415497    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2g2f7" event={"ID":"4bbbe48c-ae26-4550-af9b-77f6efdbc830","Type":"ContainerStarted","Data":"afa590d61875fa23f12f89629ccf6ac054d6282f46431ae8135e54ffccca2724"}
Jul 17 12:26:32.486274 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:32.486239    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:32.486408 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:32.486305    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:32.486408 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:32.486378    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:32.486408 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:32.486399    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 12:26:32.486541 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:32.486449    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-insights-runtime-extractor-tls podName:70772ec3-2b0c-4134-9be0-039e0a2c0dc4 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:34.486428769 +0000 UTC m=+269.490475631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ctcgp" (UID: "70772ec3-2b0c-4134-9be0-039e0a2c0dc4") : secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:32.486541 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:32.486466    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-rbac-proxy-cm podName:70772ec3-2b0c-4134-9be0-039e0a2c0dc4 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:34.486459523 +0000 UTC m=+269.490506384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ctcgp" (UID: "70772ec3-2b0c-4134-9be0-039e0a2c0dc4") : configmap "kube-rbac-proxy" not found
Jul 17 12:26:32.959698 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:32.959672    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lmgzk_460481eb-fc51-49b6-b8f7-7e892d0d0e64/dns-node-resolver/0.log"
Jul 17 12:26:33.304526 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.304499    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-2txl8"]
Jul 17 12:26:33.306248 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.306226    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-2txl8"
Jul 17 12:26:33.309381 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.309363    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-z6lns\""
Jul 17 12:26:33.309473 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.309425    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 17 12:26:33.310833 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.310812    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 17 12:26:33.310944 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.310843    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 17 12:26:33.310944 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.310861    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 17 12:26:33.315029 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.315010    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-2txl8"]
Jul 17 12:26:33.392479 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.392454    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6abdf888-160b-4413-9964-84391308e9d6-signing-key\") pod \"service-ca-58598695f7-2txl8\" (UID: \"6abdf888-160b-4413-9964-84391308e9d6\") " pod="openshift-service-ca/service-ca-58598695f7-2txl8"
Jul 17 12:26:33.392572 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.392482    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6abdf888-160b-4413-9964-84391308e9d6-signing-cabundle\") pod \"service-ca-58598695f7-2txl8\" (UID: \"6abdf888-160b-4413-9964-84391308e9d6\") " pod="openshift-service-ca/service-ca-58598695f7-2txl8"
Jul 17 12:26:33.392572 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.392498    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9njvm\" (UniqueName: \"kubernetes.io/projected/6abdf888-160b-4413-9964-84391308e9d6-kube-api-access-9njvm\") pod \"service-ca-58598695f7-2txl8\" (UID: \"6abdf888-160b-4413-9964-84391308e9d6\") " pod="openshift-service-ca/service-ca-58598695f7-2txl8"
Jul 17 12:26:33.420599 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.420569    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2g2f7" event={"ID":"4bbbe48c-ae26-4550-af9b-77f6efdbc830","Type":"ContainerStarted","Data":"a95a7fd1070ef6191a0ea6fc6e10594ff382e2d5f82548f5dfdabeb01e35ec70"}
Jul 17 12:26:33.420676 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.420615    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2g2f7" event={"ID":"4bbbe48c-ae26-4550-af9b-77f6efdbc830","Type":"ContainerStarted","Data":"bd1eb69283ad606bc3350b69fcc94087d30cb84cfd2423781bbf7d4237325abd"}
Jul 17 12:26:33.437142 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.437098    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2g2f7" podStartSLOduration=1.344717305 podStartE2EDuration="2.43708178s" totalImagesPullingTime="1.092364475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:26:33.43688136 +0000 UTC m=+268.440928241" watchObservedRunningTime="2026-07-17 12:26:33.43708178 +0000 UTC m=+268.441128662"
Jul 17 12:26:33.493391 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.493366    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6abdf888-160b-4413-9964-84391308e9d6-signing-key\") pod \"service-ca-58598695f7-2txl8\" (UID: \"6abdf888-160b-4413-9964-84391308e9d6\") " pod="openshift-service-ca/service-ca-58598695f7-2txl8"
Jul 17 12:26:33.493458 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.493394    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6abdf888-160b-4413-9964-84391308e9d6-signing-cabundle\") pod \"service-ca-58598695f7-2txl8\" (UID: \"6abdf888-160b-4413-9964-84391308e9d6\") " pod="openshift-service-ca/service-ca-58598695f7-2txl8"
Jul 17 12:26:33.493458 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.493410    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njvm\" (UniqueName: \"kubernetes.io/projected/6abdf888-160b-4413-9964-84391308e9d6-kube-api-access-9njvm\") pod \"service-ca-58598695f7-2txl8\" (UID: \"6abdf888-160b-4413-9964-84391308e9d6\") " pod="openshift-service-ca/service-ca-58598695f7-2txl8"
Jul 17 12:26:33.494243 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.494220    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6abdf888-160b-4413-9964-84391308e9d6-signing-cabundle\") pod \"service-ca-58598695f7-2txl8\" (UID: \"6abdf888-160b-4413-9964-84391308e9d6\") " pod="openshift-service-ca/service-ca-58598695f7-2txl8"
Jul 17 12:26:33.495703 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.495684    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6abdf888-160b-4413-9964-84391308e9d6-signing-key\") pod \"service-ca-58598695f7-2txl8\" (UID: \"6abdf888-160b-4413-9964-84391308e9d6\") " pod="openshift-service-ca/service-ca-58598695f7-2txl8"
Jul 17 12:26:33.502423 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.502402    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9njvm\" (UniqueName: \"kubernetes.io/projected/6abdf888-160b-4413-9964-84391308e9d6-kube-api-access-9njvm\") pod \"service-ca-58598695f7-2txl8\" (UID: \"6abdf888-160b-4413-9964-84391308e9d6\") " pod="openshift-service-ca/service-ca-58598695f7-2txl8"
Jul 17 12:26:33.614532 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.614470    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-2txl8"
Jul 17 12:26:33.735016 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.734989    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-2txl8"]
Jul 17 12:26:33.737679 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:26:33.737650    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abdf888_160b_4413_9964_84391308e9d6.slice/crio-3f732a1c4fcc928be89e67661cd7fdfe9fa2fa574ef92b43c5da38d2d2791ef1 WatchSource:0}: Error finding container 3f732a1c4fcc928be89e67661cd7fdfe9fa2fa574ef92b43c5da38d2d2791ef1: Status 404 returned error can't find the container with id 3f732a1c4fcc928be89e67661cd7fdfe9fa2fa574ef92b43c5da38d2d2791ef1
Jul 17 12:26:33.960450 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:33.960397    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q7w4l_a2bedbc3-150b-4334-a07f-574108c2901a/node-ca/0.log"
Jul 17 12:26:34.360497 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:34.360457    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4nsvg_e9ff568b-7a93-4a5d-9eac-b04e3b285830/serve-healthcheck-canary/0.log"
Jul 17 12:26:34.425725 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:34.425691    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-2txl8" event={"ID":"6abdf888-160b-4413-9964-84391308e9d6","Type":"ContainerStarted","Data":"3f732a1c4fcc928be89e67661cd7fdfe9fa2fa574ef92b43c5da38d2d2791ef1"}
Jul 17 12:26:34.502571 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:34.502540    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:34.502716 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:34.502606    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:34.502716 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:34.502693    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 12:26:34.502859 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:34.502746    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-rbac-proxy-cm podName:70772ec3-2b0c-4134-9be0-039e0a2c0dc4 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:38.502733424 +0000 UTC m=+273.506780285 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ctcgp" (UID: "70772ec3-2b0c-4134-9be0-039e0a2c0dc4") : configmap "kube-rbac-proxy" not found
Jul 17 12:26:34.502859 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:34.502693    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:34.502859 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:34.502810    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-insights-runtime-extractor-tls podName:70772ec3-2b0c-4134-9be0-039e0a2c0dc4 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:38.502798507 +0000 UTC m=+273.506845377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ctcgp" (UID: "70772ec3-2b0c-4134-9be0-039e0a2c0dc4") : secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:34.804875 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:34.804836    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e974b1cb-4c06-48c1-aded-22b183fc908b-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wcgp8\" (UID: \"e974b1cb-4c06-48c1-aded-22b183fc908b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8"
Jul 17 12:26:34.805045 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:34.804970    2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 12:26:34.805094 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:34.805054    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e974b1cb-4c06-48c1-aded-22b183fc908b-samples-operator-tls podName:e974b1cb-4c06-48c1-aded-22b183fc908b nodeName:}" failed. No retries permitted until 2026-07-17 12:26:42.80503163 +0000 UTC m=+277.809078514 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e974b1cb-4c06-48c1-aded-22b183fc908b-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-wcgp8" (UID: "e974b1cb-4c06-48c1-aded-22b183fc908b") : secret "samples-operator-tls" not found
Jul 17 12:26:36.433837 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:36.433795    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-2txl8" event={"ID":"6abdf888-160b-4413-9964-84391308e9d6","Type":"ContainerStarted","Data":"6d71e849bafdffb765effed8b602ede95ed74d4754479411b4bddfc4edf79120"}
Jul 17 12:26:36.449587 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:36.449354    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-2txl8" podStartSLOduration=1.728287066 podStartE2EDuration="3.449340009s" totalImagesPullingTime="1.721052943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:26:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:26:36.44885028 +0000 UTC m=+271.452897154" watchObservedRunningTime="2026-07-17 12:26:36.449340009 +0000 UTC m=+271.453386891"
Jul 17 12:26:38.534228 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:38.534196    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:38.534654 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:38.534278    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:38.534654 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:38.534303    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 12:26:38.534654 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:38.534362    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-rbac-proxy-cm podName:70772ec3-2b0c-4134-9be0-039e0a2c0dc4 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:46.534348345 +0000 UTC m=+281.538395206 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ctcgp" (UID: "70772ec3-2b0c-4134-9be0-039e0a2c0dc4") : configmap "kube-rbac-proxy" not found
Jul 17 12:26:38.534654 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:38.534384    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:38.534654 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:38.534429    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-insights-runtime-extractor-tls podName:70772ec3-2b0c-4134-9be0-039e0a2c0dc4 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:46.534414132 +0000 UTC m=+281.538460994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ctcgp" (UID: "70772ec3-2b0c-4134-9be0-039e0a2c0dc4") : secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:41.942551 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:41.942518    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6678b8bcb7-d4r2t"]
Jul 17 12:26:41.942923 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:41.942674    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t" podUID="c7ab02ba-f207-4ce7-896c-f1cef9abea99"
Jul 17 12:26:42.451997 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.451972    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:26:42.459256 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.459236    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:26:42.566732 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.566706    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7ab02ba-f207-4ce7-896c-f1cef9abea99-ca-trust-extracted\") pod \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") "
Jul 17 12:26:42.566866 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.566739    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7ab02ba-f207-4ce7-896c-f1cef9abea99-trusted-ca\") pod \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") "
Jul 17 12:26:42.566866 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.566781    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7ab02ba-f207-4ce7-896c-f1cef9abea99-installation-pull-secrets\") pod \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") "
Jul 17 12:26:42.566866 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.566803    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-bound-sa-token\") pod \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") "
Jul 17 12:26:42.566866 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.566834    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7ab02ba-f207-4ce7-896c-f1cef9abea99-image-registry-private-configuration\") pod \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") "
Jul 17 12:26:42.566866 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.566850    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-certificates\") pod \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") "
Jul 17 12:26:42.566866 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.566866    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5c7v\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-kube-api-access-r5c7v\") pod \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\" (UID: \"c7ab02ba-f207-4ce7-896c-f1cef9abea99\") "
Jul 17 12:26:42.567152 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.566982    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ab02ba-f207-4ce7-896c-f1cef9abea99-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c7ab02ba-f207-4ce7-896c-f1cef9abea99" (UID: "c7ab02ba-f207-4ce7-896c-f1cef9abea99"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:26:42.567222 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.567198    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ab02ba-f207-4ce7-896c-f1cef9abea99-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c7ab02ba-f207-4ce7-896c-f1cef9abea99" (UID: "c7ab02ba-f207-4ce7-896c-f1cef9abea99"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:26:42.567222 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.567205    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c7ab02ba-f207-4ce7-896c-f1cef9abea99" (UID: "c7ab02ba-f207-4ce7-896c-f1cef9abea99"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:26:42.569197 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.569167    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c7ab02ba-f207-4ce7-896c-f1cef9abea99" (UID: "c7ab02ba-f207-4ce7-896c-f1cef9abea99"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:26:42.569313 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.569217    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7ab02ba-f207-4ce7-896c-f1cef9abea99-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c7ab02ba-f207-4ce7-896c-f1cef9abea99" (UID: "c7ab02ba-f207-4ce7-896c-f1cef9abea99"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:26:42.569313 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.569264    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-kube-api-access-r5c7v" (OuterVolumeSpecName: "kube-api-access-r5c7v") pod "c7ab02ba-f207-4ce7-896c-f1cef9abea99" (UID: "c7ab02ba-f207-4ce7-896c-f1cef9abea99"). InnerVolumeSpecName "kube-api-access-r5c7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:26:42.569313 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.569293    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7ab02ba-f207-4ce7-896c-f1cef9abea99-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c7ab02ba-f207-4ce7-896c-f1cef9abea99" (UID: "c7ab02ba-f207-4ce7-896c-f1cef9abea99"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:26:42.667388 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.667360    2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7ab02ba-f207-4ce7-896c-f1cef9abea99-image-registry-private-configuration\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:26:42.667388 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.667383    2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-certificates\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:26:42.667388 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.667392    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5c7v\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-kube-api-access-r5c7v\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:26:42.667568 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.667402    2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7ab02ba-f207-4ce7-896c-f1cef9abea99-ca-trust-extracted\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:26:42.667568 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.667410    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7ab02ba-f207-4ce7-896c-f1cef9abea99-trusted-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:26:42.667568 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.667419    2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7ab02ba-f207-4ce7-896c-f1cef9abea99-installation-pull-secrets\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:26:42.667568 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.667428    2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-bound-sa-token\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:26:42.868793 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.868742    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e974b1cb-4c06-48c1-aded-22b183fc908b-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wcgp8\" (UID: \"e974b1cb-4c06-48c1-aded-22b183fc908b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8"
Jul 17 12:26:42.871187 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.871153    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e974b1cb-4c06-48c1-aded-22b183fc908b-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-wcgp8\" (UID: \"e974b1cb-4c06-48c1-aded-22b183fc908b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8"
Jul 17 12:26:42.972299 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:42.972277    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8"
Jul 17 12:26:43.091361 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:43.091340    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8"]
Jul 17 12:26:43.455849 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:43.455769    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8" event={"ID":"e974b1cb-4c06-48c1-aded-22b183fc908b","Type":"ContainerStarted","Data":"f06960b03e6ee48c36c0993337d226ff64c41af6aff9964a632eeac05dad76b3"}
Jul 17 12:26:43.455849 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:43.455785    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6678b8bcb7-d4r2t"
Jul 17 12:26:43.489804 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:43.489780    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6678b8bcb7-d4r2t"]
Jul 17 12:26:43.495361 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:43.495338    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6678b8bcb7-d4r2t"]
Jul 17 12:26:43.573495 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:43.573464    2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7ab02ba-f207-4ce7-896c-f1cef9abea99-registry-tls\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:26:43.633310 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:43.633280    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7ab02ba-f207-4ce7-896c-f1cef9abea99" path="/var/lib/kubelet/pods/c7ab02ba-f207-4ce7-896c-f1cef9abea99/volumes"
Jul 17 12:26:45.464234 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:45.464204    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8" event={"ID":"e974b1cb-4c06-48c1-aded-22b183fc908b","Type":"ContainerStarted","Data":"eec5d7bebd992cf458f06ddb51e177b5a8f9655ad014551896e31f13b5b340ed"}
Jul 17 12:26:45.464562 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:45.464280    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8" event={"ID":"e974b1cb-4c06-48c1-aded-22b183fc908b","Type":"ContainerStarted","Data":"d47b1100899c04a60b46fc745904342cde2448cca005773a4ebaf75e233c0141"}
Jul 17 12:26:45.480800 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:45.480732    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-wcgp8" podStartSLOduration=16.8600451 podStartE2EDuration="18.480718749s" totalImagesPullingTime="1.620673649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:26:45.479736606 +0000 UTC m=+280.483783498" watchObservedRunningTime="2026-07-17 12:26:45.480718749 +0000 UTC m=+280.484765609"
Jul 17 12:26:46.144044 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:26:46.143997    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mq9p4" podUID="668cc1ae-8c83-45b9-a875-b65bd76f80ea"
Jul 17 12:26:46.467429 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:46.467357    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:26:46.595518 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:46.595487    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:46.595664 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:46.595538    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:46.596133 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:46.596110    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:46.597862 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:46.597842    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/70772ec3-2b0c-4134-9be0-039e0a2c0dc4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctcgp\" (UID: \"70772ec3-2b0c-4134-9be0-039e0a2c0dc4\") " pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:46.682029 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:46.682004    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ctcgp"
Jul 17 12:26:46.804062 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:46.803985    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ctcgp"]
Jul 17 12:26:46.806673 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:26:46.806649    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70772ec3_2b0c_4134_9be0_039e0a2c0dc4.slice/crio-e3d95abe47b2ee673c10dac061d68e381f1a66365a0760708371c29e33818d6c WatchSource:0}: Error finding container e3d95abe47b2ee673c10dac061d68e381f1a66365a0760708371c29e33818d6c: Status 404 returned error can't find the container with id e3d95abe47b2ee673c10dac061d68e381f1a66365a0760708371c29e33818d6c
Jul 17 12:26:47.472583 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:47.472497    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ctcgp" event={"ID":"70772ec3-2b0c-4134-9be0-039e0a2c0dc4","Type":"ContainerStarted","Data":"e3d95abe47b2ee673c10dac061d68e381f1a66365a0760708371c29e33818d6c"}
Jul 17 12:26:49.717932 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:49.717897    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:26:49.720306 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:49.720267    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/668cc1ae-8c83-45b9-a875-b65bd76f80ea-metrics-tls\") pod \"dns-default-mq9p4\" (UID: \"668cc1ae-8c83-45b9-a875-b65bd76f80ea\") " pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:26:49.771068 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:49.771040    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gk42k\""
Jul 17 12:26:49.779333 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:49.779308    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:26:51.641689 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:51.641666    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mq9p4"]
Jul 17 12:26:51.654669 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:26:51.654646    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668cc1ae_8c83_45b9_a875_b65bd76f80ea.slice/crio-c0c42d15f3835810488f61d5f07b27b85797ab67fd50d2d379528c31590147b1 WatchSource:0}: Error finding container c0c42d15f3835810488f61d5f07b27b85797ab67fd50d2d379528c31590147b1: Status 404 returned error can't find the container with id c0c42d15f3835810488f61d5f07b27b85797ab67fd50d2d379528c31590147b1
Jul 17 12:26:52.490208 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:52.490110    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ctcgp" event={"ID":"70772ec3-2b0c-4134-9be0-039e0a2c0dc4","Type":"ContainerStarted","Data":"308b863a0f31ab86249da48cffbda2b75f7b5fc010cee857b5214dafc17625b6"}
Jul 17 12:26:52.491530 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:52.491459    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mq9p4" event={"ID":"668cc1ae-8c83-45b9-a875-b65bd76f80ea","Type":"ContainerStarted","Data":"c0c42d15f3835810488f61d5f07b27b85797ab67fd50d2d379528c31590147b1"}
Jul 17 12:26:53.534790 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:53.534086    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-jvdrq"]
Jul 17 12:26:53.541236 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:53.541208    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-jvdrq"
Jul 17 12:26:53.544035 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:53.544009    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 17 12:26:53.544143 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:53.544057    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-dwwgk\""
Jul 17 12:26:53.544143 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:53.544057    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 17 12:26:53.547745 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:53.547693    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-jvdrq"]
Jul 17 12:26:53.646639 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:53.646609    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r57tl\" (UniqueName: \"kubernetes.io/projected/7fbc6f80-0452-475d-a501-deb3474f788a-kube-api-access-r57tl\") pod \"downloads-5bc67dcd46-jvdrq\" (UID: \"7fbc6f80-0452-475d-a501-deb3474f788a\") " pod="openshift-console/downloads-5bc67dcd46-jvdrq"
Jul 17 12:26:53.746974 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:53.746945    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r57tl\" (UniqueName: \"kubernetes.io/projected/7fbc6f80-0452-475d-a501-deb3474f788a-kube-api-access-r57tl\") pod \"downloads-5bc67dcd46-jvdrq\" (UID: \"7fbc6f80-0452-475d-a501-deb3474f788a\") " pod="openshift-console/downloads-5bc67dcd46-jvdrq"
Jul 17 12:26:53.755427 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:53.755399    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r57tl\" (UniqueName: \"kubernetes.io/projected/7fbc6f80-0452-475d-a501-deb3474f788a-kube-api-access-r57tl\") pod \"downloads-5bc67dcd46-jvdrq\" (UID: \"7fbc6f80-0452-475d-a501-deb3474f788a\") " pod="openshift-console/downloads-5bc67dcd46-jvdrq"
Jul 17 12:26:53.851759 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:53.851668    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-jvdrq"
Jul 17 12:26:55.435710 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:55.435678    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-jvdrq"]
Jul 17 12:26:55.439341 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:26:55.439310    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbc6f80_0452_475d_a501_deb3474f788a.slice/crio-18972faff31df0ae428a73bf0dbe5b57cae539152b6f0d2fd6278d055a245888 WatchSource:0}: Error finding container 18972faff31df0ae428a73bf0dbe5b57cae539152b6f0d2fd6278d055a245888: Status 404 returned error can't find the container with id 18972faff31df0ae428a73bf0dbe5b57cae539152b6f0d2fd6278d055a245888
Jul 17 12:26:55.505109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:55.505084    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-jvdrq" event={"ID":"7fbc6f80-0452-475d-a501-deb3474f788a","Type":"ContainerStarted","Data":"18972faff31df0ae428a73bf0dbe5b57cae539152b6f0d2fd6278d055a245888"}
Jul 17 12:26:55.507271 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:55.507250    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ctcgp" event={"ID":"70772ec3-2b0c-4134-9be0-039e0a2c0dc4","Type":"ContainerStarted","Data":"1abaa9a0180c21c497aae246a0540e10b49b8d244f96ef8aa1e640a9ba55ff39"}
Jul 17 12:26:55.509538 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:55.509019    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mq9p4" event={"ID":"668cc1ae-8c83-45b9-a875-b65bd76f80ea","Type":"ContainerStarted","Data":"ba4930af684a1e39ae421a9b69b9246e50c11155b3b0d508f96ac48af2f7e362"}
Jul 17 12:26:55.509538 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:55.509065    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mq9p4" event={"ID":"668cc1ae-8c83-45b9-a875-b65bd76f80ea","Type":"ContainerStarted","Data":"8a5f396db010539a9c20f815e5381c3883ecd377106d1024bb02880f7090b20d"}
Jul 17 12:26:55.509538 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:55.509176    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:26:55.524977 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:55.524941    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mq9p4" podStartSLOduration=252.877577652 podStartE2EDuration="4m16.52492663s" totalImagesPullingTime="3.647348978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:22:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:26:55.524894869 +0000 UTC m=+290.528941750" watchObservedRunningTime="2026-07-17 12:26:55.52492663 +0000 UTC m=+290.528973511"
Jul 17 12:26:58.847857 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:58.847790    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c4665b67d-kc9q2"]
Jul 17 12:26:58.871054 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:58.871015    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c4665b67d-kc9q2"]
Jul 17 12:26:58.871213 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:58.871072    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:58.874309 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:58.874087    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 17 12:26:58.874309 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:58.874126    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 17 12:26:58.874309 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:58.874176    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 17 12:26:58.874611 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:58.874352    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 17 12:26:58.875164 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:58.875140    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-28frs\""
Jul 17 12:26:58.875306 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:58.875144    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 17 12:26:58.987646 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:58.987613    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-console-serving-cert\") pod \"console-7c4665b67d-kc9q2\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") " pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:58.987834 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:58.987654    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-oauth-serving-cert\") pod \"console-7c4665b67d-kc9q2\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") " pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:58.987834 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:58.987732    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-service-ca\") pod \"console-7c4665b67d-kc9q2\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") " pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:58.987834 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:58.987814    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-console-config\") pod \"console-7c4665b67d-kc9q2\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") " pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:58.987992 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:58.987851    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-console-oauth-config\") pod \"console-7c4665b67d-kc9q2\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") " pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:58.987992 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:58.987906    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdd8z\" (UniqueName: \"kubernetes.io/projected/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-kube-api-access-vdd8z\") pod \"console-7c4665b67d-kc9q2\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") " pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:59.088985 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:59.088951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-console-config\") pod \"console-7c4665b67d-kc9q2\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") " pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:59.089141 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:59.088997    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-console-oauth-config\") pod \"console-7c4665b67d-kc9q2\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") " pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:59.089141 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:59.089023    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdd8z\" (UniqueName: \"kubernetes.io/projected/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-kube-api-access-vdd8z\") pod \"console-7c4665b67d-kc9q2\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") " pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:59.090038 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:59.089245    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-console-serving-cert\") pod \"console-7c4665b67d-kc9q2\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") " pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:59.090038 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:59.089286    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-oauth-serving-cert\") pod \"console-7c4665b67d-kc9q2\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") " pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:59.090038 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:59.089323    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-service-ca\") pod \"console-7c4665b67d-kc9q2\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") " pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:59.090038 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:59.089742    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-console-config\") pod \"console-7c4665b67d-kc9q2\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") " pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:59.090038 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:59.089969    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-service-ca\") pod \"console-7c4665b67d-kc9q2\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") " pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:59.090657 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:59.090614    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-oauth-serving-cert\") pod \"console-7c4665b67d-kc9q2\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") " pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:59.092249 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:59.092210    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-console-serving-cert\") pod \"console-7c4665b67d-kc9q2\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") " pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:59.092371 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:59.092286    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-console-oauth-config\") pod \"console-7c4665b67d-kc9q2\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") " pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:59.097288 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:59.097253    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdd8z\" (UniqueName: \"kubernetes.io/projected/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-kube-api-access-vdd8z\") pod \"console-7c4665b67d-kc9q2\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") " pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:59.183315 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:59.183234    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:26:59.328118 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:59.328094    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c4665b67d-kc9q2"]
Jul 17 12:26:59.331356 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:26:59.331330    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fdd766d_edc4_4aa8_96c9_e0c693d27ce8.slice/crio-1081e55899bc36b06fc3f444937251eecc292c35147cb73fcfbefd76832ae80f WatchSource:0}: Error finding container 1081e55899bc36b06fc3f444937251eecc292c35147cb73fcfbefd76832ae80f: Status 404 returned error can't find the container with id 1081e55899bc36b06fc3f444937251eecc292c35147cb73fcfbefd76832ae80f
Jul 17 12:26:59.526712 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:59.526683    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4665b67d-kc9q2" event={"ID":"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8","Type":"ContainerStarted","Data":"1081e55899bc36b06fc3f444937251eecc292c35147cb73fcfbefd76832ae80f"}
Jul 17 12:26:59.528979 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:59.528952    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ctcgp" event={"ID":"70772ec3-2b0c-4134-9be0-039e0a2c0dc4","Type":"ContainerStarted","Data":"0c57aae90f653077793c6f379bc188da03d71125efe2417b2af1594901a0cf82"}
Jul 17 12:26:59.546573 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:26:59.546529    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ctcgp" podStartSLOduration=17.235735747 podStartE2EDuration="29.546512472s" totalImagesPullingTime="12.310776725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:26:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:26:59.545555517 +0000 UTC m=+294.549602411" watchObservedRunningTime="2026-07-17 12:26:59.546512472 +0000 UTC m=+294.550559331"
Jul 17 12:27:00.629802 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:27:00.629724    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-4nbwg" podUID="966fc2f5-1448-4a14-8b27-21e714628e1a"
Jul 17 12:27:02.542416 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:02.542337    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4665b67d-kc9q2" event={"ID":"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8","Type":"ContainerStarted","Data":"433cd6e574e546ba2cc6c831b848467f500607c29e795ea688b860dfa1b13fd7"}
Jul 17 12:27:02.560675 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:02.560626    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c4665b67d-kc9q2" podStartSLOduration=1.641574525 podStartE2EDuration="4.560612128s" totalImagesPullingTime="2.919037603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:02.559560446 +0000 UTC m=+297.563607331" watchObservedRunningTime="2026-07-17 12:27:02.560612128 +0000 UTC m=+297.564658991"
Jul 17 12:27:03.527490 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.527440    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-9cvww"]
Jul 17 12:27:03.531097 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.531067    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww"
Jul 17 12:27:03.534056 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.533990    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 17 12:27:03.534209 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.534115    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 12:27:03.534209 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.534127    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 12:27:03.534327 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.534243    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 17 12:27:03.535603 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.535575    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-blpql\""
Jul 17 12:27:03.535718 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.535593    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 12:27:03.538672 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.538642    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-9cvww"]
Jul 17 12:27:03.628185 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.628148    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1810c267-db7d-4336-b21c-cde97c4de6e1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-9cvww\" (UID: \"1810c267-db7d-4336-b21c-cde97c4de6e1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww"
Jul 17 12:27:03.628661 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.628226    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1810c267-db7d-4336-b21c-cde97c4de6e1-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-9cvww\" (UID: \"1810c267-db7d-4336-b21c-cde97c4de6e1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww"
Jul 17 12:27:03.628661 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.628251    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2shd\" (UniqueName: \"kubernetes.io/projected/1810c267-db7d-4336-b21c-cde97c4de6e1-kube-api-access-f2shd\") pod \"prometheus-operator-5945b5cfd-9cvww\" (UID: \"1810c267-db7d-4336-b21c-cde97c4de6e1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww"
Jul 17 12:27:03.628661 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.628328    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1810c267-db7d-4336-b21c-cde97c4de6e1-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-9cvww\" (UID: \"1810c267-db7d-4336-b21c-cde97c4de6e1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww"
Jul 17 12:27:03.729013 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.728975    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1810c267-db7d-4336-b21c-cde97c4de6e1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-9cvww\" (UID: \"1810c267-db7d-4336-b21c-cde97c4de6e1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww"
Jul 17 12:27:03.729189 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.729038    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1810c267-db7d-4336-b21c-cde97c4de6e1-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-9cvww\" (UID: \"1810c267-db7d-4336-b21c-cde97c4de6e1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww"
Jul 17 12:27:03.729189 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.729146    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2shd\" (UniqueName: \"kubernetes.io/projected/1810c267-db7d-4336-b21c-cde97c4de6e1-kube-api-access-f2shd\") pod \"prometheus-operator-5945b5cfd-9cvww\" (UID: \"1810c267-db7d-4336-b21c-cde97c4de6e1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww"
Jul 17 12:27:03.729302 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.729214    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1810c267-db7d-4336-b21c-cde97c4de6e1-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-9cvww\" (UID: \"1810c267-db7d-4336-b21c-cde97c4de6e1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww"
Jul 17 12:27:03.729366 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:27:03.729339    2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 17 12:27:03.729415 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:27:03.729394    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1810c267-db7d-4336-b21c-cde97c4de6e1-prometheus-operator-tls podName:1810c267-db7d-4336-b21c-cde97c4de6e1 nodeName:}" failed. No retries permitted until 2026-07-17 12:27:04.229375633 +0000 UTC m=+299.233422506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/1810c267-db7d-4336-b21c-cde97c4de6e1-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-9cvww" (UID: "1810c267-db7d-4336-b21c-cde97c4de6e1") : secret "prometheus-operator-tls" not found
Jul 17 12:27:03.729843 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.729821    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1810c267-db7d-4336-b21c-cde97c4de6e1-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-9cvww\" (UID: \"1810c267-db7d-4336-b21c-cde97c4de6e1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww"
Jul 17 12:27:03.731961 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.731936    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1810c267-db7d-4336-b21c-cde97c4de6e1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-9cvww\" (UID: \"1810c267-db7d-4336-b21c-cde97c4de6e1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww"
Jul 17 12:27:03.739829 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:03.739809    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2shd\" (UniqueName: \"kubernetes.io/projected/1810c267-db7d-4336-b21c-cde97c4de6e1-kube-api-access-f2shd\") pod \"prometheus-operator-5945b5cfd-9cvww\" (UID: \"1810c267-db7d-4336-b21c-cde97c4de6e1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww"
Jul 17 12:27:04.232279 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:04.232247    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1810c267-db7d-4336-b21c-cde97c4de6e1-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-9cvww\" (UID: \"1810c267-db7d-4336-b21c-cde97c4de6e1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww"
Jul 17 12:27:04.234798 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:04.234774    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1810c267-db7d-4336-b21c-cde97c4de6e1-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-9cvww\" (UID: \"1810c267-db7d-4336-b21c-cde97c4de6e1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww"
Jul 17 12:27:04.442867 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:04.442833    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww"
Jul 17 12:27:04.577963 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:04.577939    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-9cvww"]
Jul 17 12:27:04.580471 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:27:04.580441    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1810c267_db7d_4336_b21c_cde97c4de6e1.slice/crio-581936f3af7be10fa5f886d88b1c92e26e9be61f7d4893a8828340f9ea7acd3c WatchSource:0}: Error finding container 581936f3af7be10fa5f886d88b1c92e26e9be61f7d4893a8828340f9ea7acd3c: Status 404 returned error can't find the container with id 581936f3af7be10fa5f886d88b1c92e26e9be61f7d4893a8828340f9ea7acd3c
Jul 17 12:27:05.515624 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:05.515590    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mq9p4"
Jul 17 12:27:05.554597 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:05.554565    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww" event={"ID":"1810c267-db7d-4336-b21c-cde97c4de6e1","Type":"ContainerStarted","Data":"581936f3af7be10fa5f886d88b1c92e26e9be61f7d4893a8828340f9ea7acd3c"}
Jul 17 12:27:09.183490 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:09.183450    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:27:09.189963 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:09.183509    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:27:09.189963 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:09.189902    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:27:09.572867 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:09.572838    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:27:13.174400 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:13.174374    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:27:13.174901 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:13.174883    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:27:13.178391 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:13.178374    2569 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 12:27:13.585840 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:13.585577    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww" event={"ID":"1810c267-db7d-4336-b21c-cde97c4de6e1","Type":"ContainerStarted","Data":"94ca5826dca99ad3b50990eb1d4d53957684993a1c710f45e3df1767a683181c"}
Jul 17 12:27:13.585840 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:13.585681    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww" event={"ID":"1810c267-db7d-4336-b21c-cde97c4de6e1","Type":"ContainerStarted","Data":"5427d032d600acc468c3b8b755c1a0ac0718f15337dbc2292ecf11baa189e363"}
Jul 17 12:27:13.587259 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:13.587230    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-jvdrq" event={"ID":"7fbc6f80-0452-475d-a501-deb3474f788a","Type":"ContainerStarted","Data":"4e5e573f8db47ee9f21068646f162c91d000c7cf2ab6458aaf5346c3e4c479e9"}
Jul 17 12:27:13.587474 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:13.587457    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-jvdrq"
Jul 17 12:27:13.598599 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:13.598575    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-jvdrq"
Jul 17 12:27:13.603814 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:13.603762    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9cvww" podStartSLOduration=2.024680102 podStartE2EDuration="10.603731957s" totalImagesPullingTime="8.579051855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:13.602521672 +0000 UTC m=+308.606568554" watchObservedRunningTime="2026-07-17 12:27:13.603731957 +0000 UTC m=+308.607778839"
Jul 17 12:27:13.620176 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:13.620130    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-jvdrq" podStartSLOduration=2.873992129 podStartE2EDuration="20.620116843s" totalImagesPullingTime="17.746124714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:13.619397789 +0000 UTC m=+308.623444673" watchObservedRunningTime="2026-07-17 12:27:13.620116843 +0000 UTC m=+308.624163725"
Jul 17 12:27:15.630364 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:15.630335    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:27:15.861453 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:15.861413    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"]
Jul 17 12:27:15.888092 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:15.888018    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"
Jul 17 12:27:15.888475 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:15.887909    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2j4mj"]
Jul 17 12:27:15.898801 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:15.898696    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 17 12:27:15.905769 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:15.899170    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 17 12:27:15.905769 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:15.900358    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-sm5v7\""
Jul 17 12:27:15.909450 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:15.909218    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"]
Jul 17 12:27:15.909450 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:15.909345    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:15.912203 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:15.912181    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 12:27:15.912410 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:15.912389    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 12:27:15.912908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:15.912888    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 12:27:15.913087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:15.913072    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rsxtw\""
Jul 17 12:27:15.931436 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:15.931413    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59ba81f3-2110-48a1-9079-90e55119c423-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-ntjbj\" (UID: \"59ba81f3-2110-48a1-9079-90e55119c423\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"
Jul 17 12:27:15.931607 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:15.931588    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59ba81f3-2110-48a1-9079-90e55119c423-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-ntjbj\" (UID: \"59ba81f3-2110-48a1-9079-90e55119c423\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"
Jul 17 12:27:15.931791 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:15.931771    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/59ba81f3-2110-48a1-9079-90e55119c423-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ntjbj\" (UID: \"59ba81f3-2110-48a1-9079-90e55119c423\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"
Jul 17 12:27:15.931996 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:15.931973    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkn79\" (UniqueName: \"kubernetes.io/projected/59ba81f3-2110-48a1-9079-90e55119c423-kube-api-access-xkn79\") pod \"openshift-state-metrics-79f6958cdf-ntjbj\" (UID: \"59ba81f3-2110-48a1-9079-90e55119c423\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"
Jul 17 12:27:16.032664 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.032635    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59ba81f3-2110-48a1-9079-90e55119c423-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-ntjbj\" (UID: \"59ba81f3-2110-48a1-9079-90e55119c423\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"
Jul 17 12:27:16.032816 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.032683    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bcff7d42-41a1-45d3-b58a-e11ac6c40173-node-exporter-accelerators-collector-config\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.032816 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.032711    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/59ba81f3-2110-48a1-9079-90e55119c423-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ntjbj\" (UID: \"59ba81f3-2110-48a1-9079-90e55119c423\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"
Jul 17 12:27:16.032816 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.032789    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bcff7d42-41a1-45d3-b58a-e11ac6c40173-node-exporter-textfile\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.032992 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.032828    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bcff7d42-41a1-45d3-b58a-e11ac6c40173-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.032992 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.032852    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bcff7d42-41a1-45d3-b58a-e11ac6c40173-root\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.032992 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.032897    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bcff7d42-41a1-45d3-b58a-e11ac6c40173-node-exporter-tls\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.032992 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.032924    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt5pr\" (UniqueName: \"kubernetes.io/projected/bcff7d42-41a1-45d3-b58a-e11ac6c40173-kube-api-access-dt5pr\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.032992 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.032953    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcff7d42-41a1-45d3-b58a-e11ac6c40173-metrics-client-ca\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.033226 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.032993    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcff7d42-41a1-45d3-b58a-e11ac6c40173-sys\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.033226 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.033028    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkn79\" (UniqueName: \"kubernetes.io/projected/59ba81f3-2110-48a1-9079-90e55119c423-kube-api-access-xkn79\") pod \"openshift-state-metrics-79f6958cdf-ntjbj\" (UID: \"59ba81f3-2110-48a1-9079-90e55119c423\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"
Jul 17 12:27:16.033226 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.033079    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59ba81f3-2110-48a1-9079-90e55119c423-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-ntjbj\" (UID: \"59ba81f3-2110-48a1-9079-90e55119c423\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"
Jul 17 12:27:16.033374 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:27:16.033300    2569 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 17 12:27:16.033374 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:27:16.033362    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ba81f3-2110-48a1-9079-90e55119c423-openshift-state-metrics-tls podName:59ba81f3-2110-48a1-9079-90e55119c423 nodeName:}" failed. No retries permitted until 2026-07-17 12:27:16.533342754 +0000 UTC m=+311.537389626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/59ba81f3-2110-48a1-9079-90e55119c423-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-ntjbj" (UID: "59ba81f3-2110-48a1-9079-90e55119c423") : secret "openshift-state-metrics-tls" not found
Jul 17 12:27:16.033903 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.033881    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59ba81f3-2110-48a1-9079-90e55119c423-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-ntjbj\" (UID: \"59ba81f3-2110-48a1-9079-90e55119c423\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"
Jul 17 12:27:16.034080 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.034063    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bcff7d42-41a1-45d3-b58a-e11ac6c40173-node-exporter-wtmp\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.035918 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.035896    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59ba81f3-2110-48a1-9079-90e55119c423-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-ntjbj\" (UID: \"59ba81f3-2110-48a1-9079-90e55119c423\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"
Jul 17 12:27:16.044488 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.044465    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkn79\" (UniqueName: \"kubernetes.io/projected/59ba81f3-2110-48a1-9079-90e55119c423-kube-api-access-xkn79\") pod \"openshift-state-metrics-79f6958cdf-ntjbj\" (UID: \"59ba81f3-2110-48a1-9079-90e55119c423\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"
Jul 17 12:27:16.135678 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.135643    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bcff7d42-41a1-45d3-b58a-e11ac6c40173-node-exporter-textfile\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.135900 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.135883    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bcff7d42-41a1-45d3-b58a-e11ac6c40173-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.136016 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.136003    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bcff7d42-41a1-45d3-b58a-e11ac6c40173-root\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.136125 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.136113    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bcff7d42-41a1-45d3-b58a-e11ac6c40173-node-exporter-tls\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.136230 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.136216    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt5pr\" (UniqueName: \"kubernetes.io/projected/bcff7d42-41a1-45d3-b58a-e11ac6c40173-kube-api-access-dt5pr\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.136338 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.136326    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcff7d42-41a1-45d3-b58a-e11ac6c40173-metrics-client-ca\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.136462 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.136450    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcff7d42-41a1-45d3-b58a-e11ac6c40173-sys\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.136580 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.136569    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bcff7d42-41a1-45d3-b58a-e11ac6c40173-node-exporter-wtmp\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.136706 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.136693    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bcff7d42-41a1-45d3-b58a-e11ac6c40173-node-exporter-accelerators-collector-config\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.137519 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.137501    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bcff7d42-41a1-45d3-b58a-e11ac6c40173-node-exporter-accelerators-collector-config\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.137847 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.137830    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcff7d42-41a1-45d3-b58a-e11ac6c40173-sys\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.138023 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.137945    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bcff7d42-41a1-45d3-b58a-e11ac6c40173-node-exporter-wtmp\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.138180 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.138142    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bcff7d42-41a1-45d3-b58a-e11ac6c40173-root\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.138404 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.138380    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcff7d42-41a1-45d3-b58a-e11ac6c40173-metrics-client-ca\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.138647 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:27:16.138599    2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 17 12:27:16.138805 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:27:16.138795    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcff7d42-41a1-45d3-b58a-e11ac6c40173-node-exporter-tls podName:bcff7d42-41a1-45d3-b58a-e11ac6c40173 nodeName:}" failed. No retries permitted until 2026-07-17 12:27:16.638775531 +0000 UTC m=+311.642822408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/bcff7d42-41a1-45d3-b58a-e11ac6c40173-node-exporter-tls") pod "node-exporter-2j4mj" (UID: "bcff7d42-41a1-45d3-b58a-e11ac6c40173") : secret "node-exporter-tls" not found
Jul 17 12:27:16.139134 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.139104    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bcff7d42-41a1-45d3-b58a-e11ac6c40173-node-exporter-textfile\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.140988 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.140965    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bcff7d42-41a1-45d3-b58a-e11ac6c40173-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.151210 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.151185    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt5pr\" (UniqueName: \"kubernetes.io/projected/bcff7d42-41a1-45d3-b58a-e11ac6c40173-kube-api-access-dt5pr\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.541133 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.541093    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/59ba81f3-2110-48a1-9079-90e55119c423-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ntjbj\" (UID: \"59ba81f3-2110-48a1-9079-90e55119c423\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"
Jul 17 12:27:16.541312 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:27:16.541296    2569 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 17 12:27:16.541383 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:27:16.541360    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ba81f3-2110-48a1-9079-90e55119c423-openshift-state-metrics-tls podName:59ba81f3-2110-48a1-9079-90e55119c423 nodeName:}" failed. No retries permitted until 2026-07-17 12:27:17.541340976 +0000 UTC m=+312.545387840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/59ba81f3-2110-48a1-9079-90e55119c423-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-ntjbj" (UID: "59ba81f3-2110-48a1-9079-90e55119c423") : secret "openshift-state-metrics-tls" not found
Jul 17 12:27:16.641804 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.641769    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bcff7d42-41a1-45d3-b58a-e11ac6c40173-node-exporter-tls\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.644534 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.644511    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bcff7d42-41a1-45d3-b58a-e11ac6c40173-node-exporter-tls\") pod \"node-exporter-2j4mj\" (UID: \"bcff7d42-41a1-45d3-b58a-e11ac6c40173\") " pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.823310 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.823191    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2j4mj"
Jul 17 12:27:16.861872 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:27:16.861844    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcff7d42_41a1_45d3_b58a_e11ac6c40173.slice/crio-4488ac6594fe7b627942d257a44013960f96eb4600355163b41828c01aaeab77 WatchSource:0}: Error finding container 4488ac6594fe7b627942d257a44013960f96eb4600355163b41828c01aaeab77: Status 404 returned error can't find the container with id 4488ac6594fe7b627942d257a44013960f96eb4600355163b41828c01aaeab77
Jul 17 12:27:16.863790 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:16.863771    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 12:27:17.548762 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:17.548715    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/59ba81f3-2110-48a1-9079-90e55119c423-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ntjbj\" (UID: \"59ba81f3-2110-48a1-9079-90e55119c423\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"
Jul 17 12:27:17.552132 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:17.552105    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/59ba81f3-2110-48a1-9079-90e55119c423-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ntjbj\" (UID: \"59ba81f3-2110-48a1-9079-90e55119c423\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"
Jul 17 12:27:17.606970 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:17.606937    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2j4mj" event={"ID":"bcff7d42-41a1-45d3-b58a-e11ac6c40173","Type":"ContainerStarted","Data":"4488ac6594fe7b627942d257a44013960f96eb4600355163b41828c01aaeab77"}
Jul 17 12:27:17.709700 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:17.709670    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"
Jul 17 12:27:18.110677 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:18.110642    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj"]
Jul 17 12:27:18.113946 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:27:18.113911    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ba81f3_2110_48a1_9079_90e55119c423.slice/crio-afeee2c93ad7fff8d6519555c0a4a40fe335d8833e7315c12db9f9a07b7e98bc WatchSource:0}: Error finding container afeee2c93ad7fff8d6519555c0a4a40fe335d8833e7315c12db9f9a07b7e98bc: Status 404 returned error can't find the container with id afeee2c93ad7fff8d6519555c0a4a40fe335d8833e7315c12db9f9a07b7e98bc
Jul 17 12:27:18.568009 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:18.567931    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c4665b67d-kc9q2"]
Jul 17 12:27:18.612068 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:18.612038    2569 generic.go:356] "Generic (PLEG): container finished" podID="bcff7d42-41a1-45d3-b58a-e11ac6c40173" containerID="03e8466ba2d36e9d401487b967a06e3260f8ccab02d7321b2d98d2b9f6948ae3" exitCode=0
Jul 17 12:27:18.612186 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:18.612124    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2j4mj" event={"ID":"bcff7d42-41a1-45d3-b58a-e11ac6c40173","Type":"ContainerDied","Data":"03e8466ba2d36e9d401487b967a06e3260f8ccab02d7321b2d98d2b9f6948ae3"}
Jul 17 12:27:18.614436 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:18.614415    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj" event={"ID":"59ba81f3-2110-48a1-9079-90e55119c423","Type":"ContainerStarted","Data":"07795eb209c3cd48b23d5d0d1204dd278b8ac913610d2ef2c8e57b809093e5d4"}
Jul 17 12:27:18.614542 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:18.614467    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj" event={"ID":"59ba81f3-2110-48a1-9079-90e55119c423","Type":"ContainerStarted","Data":"c70b007edb04c3489620d146c59551f57ba98254f547d775e8ba00dae04e2073"}
Jul 17 12:27:18.614542 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:18.614500    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj" event={"ID":"59ba81f3-2110-48a1-9079-90e55119c423","Type":"ContainerStarted","Data":"afeee2c93ad7fff8d6519555c0a4a40fe335d8833e7315c12db9f9a07b7e98bc"}
Jul 17 12:27:19.620651 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:19.620620    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2j4mj" event={"ID":"bcff7d42-41a1-45d3-b58a-e11ac6c40173","Type":"ContainerStarted","Data":"53cfab799b484042a8bdcaffbf358715c1c73e8c89e52b6e96ef26a03a1a56d7"}
Jul 17 12:27:19.621111 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:19.620682    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2j4mj" event={"ID":"bcff7d42-41a1-45d3-b58a-e11ac6c40173","Type":"ContainerStarted","Data":"e59b64542e17292920ab64fbf1c2ef95f51ef590cd49706473f9fb0d9df70653"}
Jul 17 12:27:19.665276 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:19.665223    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2j4mj" podStartSLOduration=3.095144838 podStartE2EDuration="4.665205066s" totalImagesPullingTime="1.446633418s" totalInitContainerRuntime="123.42681ms" isStatefulPod=true podCreationTimestamp="2026-07-17 12:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:19.663582314 +0000 UTC m=+314.667629195" watchObservedRunningTime="2026-07-17 12:27:19.665205066 +0000 UTC m=+314.669251948"
Jul 17 12:27:20.176746 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.176707    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-85cbb9955f-4hzpd"]
Jul 17 12:27:20.197102 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.197073    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85cbb9955f-4hzpd"]
Jul 17 12:27:20.197227 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.197190    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.200112 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.200090    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 17 12:27:20.200112 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.200105    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4knf0hogordj5\""
Jul 17 12:27:20.200284 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.200142    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 17 12:27:20.200473 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.200407    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 17 12:27:20.201353 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.201335    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 17 12:27:20.201440 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.201391    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nwwlh\""
Jul 17 12:27:20.271936 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.271909    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fcb3eb8a-314a-44d7-a076-5267528a012c-audit-log\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.272044 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.271947    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fcb3eb8a-314a-44d7-a076-5267528a012c-client-ca-bundle\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.272044 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.271973    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fcb3eb8a-314a-44d7-a076-5267528a012c-secret-metrics-server-tls\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.272044 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.271997    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxzr4\" (UniqueName: \"kubernetes.io/projected/fcb3eb8a-314a-44d7-a076-5267528a012c-kube-api-access-xxzr4\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.272173 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.272062    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fcb3eb8a-314a-44d7-a076-5267528a012c-metrics-server-audit-profiles\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.272173 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.272098    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcb3eb8a-314a-44d7-a076-5267528a012c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.272173 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.272152    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fcb3eb8a-314a-44d7-a076-5267528a012c-secret-metrics-server-client-certs\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.372723 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.372657    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fcb3eb8a-314a-44d7-a076-5267528a012c-audit-log\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.372723 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.372695    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fcb3eb8a-314a-44d7-a076-5267528a012c-client-ca-bundle\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.372723 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.372714    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fcb3eb8a-314a-44d7-a076-5267528a012c-secret-metrics-server-tls\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.372952 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.372764    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxzr4\" (UniqueName: \"kubernetes.io/projected/fcb3eb8a-314a-44d7-a076-5267528a012c-kube-api-access-xxzr4\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.373079 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.373049    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fcb3eb8a-314a-44d7-a076-5267528a012c-metrics-server-audit-profiles\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.373204 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.373104    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcb3eb8a-314a-44d7-a076-5267528a012c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.373204 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.373122    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fcb3eb8a-314a-44d7-a076-5267528a012c-audit-log\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.373204 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.373169    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fcb3eb8a-314a-44d7-a076-5267528a012c-secret-metrics-server-client-certs\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.373818 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.373790    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcb3eb8a-314a-44d7-a076-5267528a012c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.374002 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.373980    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fcb3eb8a-314a-44d7-a076-5267528a012c-metrics-server-audit-profiles\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.375557 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.375530    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fcb3eb8a-314a-44d7-a076-5267528a012c-secret-metrics-server-tls\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.375658 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.375575    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fcb3eb8a-314a-44d7-a076-5267528a012c-client-ca-bundle\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.375799 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.375776    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fcb3eb8a-314a-44d7-a076-5267528a012c-secret-metrics-server-client-certs\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.381510 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.381491    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxzr4\" (UniqueName: \"kubernetes.io/projected/fcb3eb8a-314a-44d7-a076-5267528a012c-kube-api-access-xxzr4\") pod \"metrics-server-85cbb9955f-4hzpd\" (UID: \"fcb3eb8a-314a-44d7-a076-5267528a012c\") " pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.509087 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.509063    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:20.629477 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.628996    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj" event={"ID":"59ba81f3-2110-48a1-9079-90e55119c423","Type":"ContainerStarted","Data":"b6438e757b4fa6f8218ad8930ea6b95ba5547e3ce385590f2895cdff4b64d68d"}
Jul 17 12:27:20.648768 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.648590    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ntjbj" podStartSLOduration=4.064341954 podStartE2EDuration="5.6485683s" totalImagesPullingTime="1.584226346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:20.645048601 +0000 UTC m=+315.649095475" watchObservedRunningTime="2026-07-17 12:27:20.6485683 +0000 UTC m=+315.652615183"
Jul 17 12:27:20.687136 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:20.687103    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85cbb9955f-4hzpd"]
Jul 17 12:27:20.691739 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:27:20.691506    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb3eb8a_314a_44d7_a076_5267528a012c.slice/crio-2c9c2966bf732827ac44c0c7645a63735c2c958baa28e0864578978ec84e7ffa WatchSource:0}: Error finding container 2c9c2966bf732827ac44c0c7645a63735c2c958baa28e0864578978ec84e7ffa: Status 404 returned error can't find the container with id 2c9c2966bf732827ac44c0c7645a63735c2c958baa28e0864578978ec84e7ffa
Jul 17 12:27:21.049897 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.049859    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-587b64d868-wpsz7"]
Jul 17 12:27:21.075930 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.075906    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-587b64d868-wpsz7"]
Jul 17 12:27:21.076074 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.076034    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.079064 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.078738    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 17 12:27:21.079064 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.078939    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 17 12:27:21.079064 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.079025    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ds67p\""
Jul 17 12:27:21.079306 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.079103    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 17 12:27:21.079361 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.079334    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 17 12:27:21.079412 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.079370    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 17 12:27:21.084256 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.084231    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 17 12:27:21.179618 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.179586    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/493d99ab-0d82-456d-b003-964aec113cc4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.179771 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.179633    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/493d99ab-0d82-456d-b003-964aec113cc4-metrics-client-ca\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.179771 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.179686    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/493d99ab-0d82-456d-b003-964aec113cc4-federate-client-tls\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.179771 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.179719    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/493d99ab-0d82-456d-b003-964aec113cc4-secret-telemeter-client\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.179943 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.179779    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/493d99ab-0d82-456d-b003-964aec113cc4-serving-certs-ca-bundle\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.179943 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.179807    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6wlb\" (UniqueName: \"kubernetes.io/projected/493d99ab-0d82-456d-b003-964aec113cc4-kube-api-access-z6wlb\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.179943 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.179859    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/493d99ab-0d82-456d-b003-964aec113cc4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.180062 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.179941    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/493d99ab-0d82-456d-b003-964aec113cc4-telemeter-client-tls\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.280997 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.280968    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/493d99ab-0d82-456d-b003-964aec113cc4-metrics-client-ca\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.281196 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.281047    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/493d99ab-0d82-456d-b003-964aec113cc4-federate-client-tls\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.281196 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.281123    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/493d99ab-0d82-456d-b003-964aec113cc4-secret-telemeter-client\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.281196 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.281165    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/493d99ab-0d82-456d-b003-964aec113cc4-serving-certs-ca-bundle\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.281361 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.281203    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6wlb\" (UniqueName: \"kubernetes.io/projected/493d99ab-0d82-456d-b003-964aec113cc4-kube-api-access-z6wlb\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.281361 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.281255    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/493d99ab-0d82-456d-b003-964aec113cc4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.281361 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.281308    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/493d99ab-0d82-456d-b003-964aec113cc4-telemeter-client-tls\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.281530 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.281360    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/493d99ab-0d82-456d-b003-964aec113cc4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.281798 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.281742    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/493d99ab-0d82-456d-b003-964aec113cc4-metrics-client-ca\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.282484 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.282457    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/493d99ab-0d82-456d-b003-964aec113cc4-serving-certs-ca-bundle\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.283434 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.283409    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/493d99ab-0d82-456d-b003-964aec113cc4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.284395 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.284369    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/493d99ab-0d82-456d-b003-964aec113cc4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.285273 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.285251    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/493d99ab-0d82-456d-b003-964aec113cc4-secret-telemeter-client\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.285365 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.285278    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/493d99ab-0d82-456d-b003-964aec113cc4-federate-client-tls\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.285605 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.285566    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/493d99ab-0d82-456d-b003-964aec113cc4-telemeter-client-tls\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.290932 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.290892    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6wlb\" (UniqueName: \"kubernetes.io/projected/493d99ab-0d82-456d-b003-964aec113cc4-kube-api-access-z6wlb\") pod \"telemeter-client-587b64d868-wpsz7\" (UID: \"493d99ab-0d82-456d-b003-964aec113cc4\") " pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.387662 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.387607    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7"
Jul 17 12:27:21.575502 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.575345    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-587b64d868-wpsz7"]
Jul 17 12:27:21.580546 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:27:21.580505    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493d99ab_0d82_456d_b003_964aec113cc4.slice/crio-f66822ecb88b6835c12bf4422bef94f19165d896806b66f85eff09dd12603c56 WatchSource:0}: Error finding container f66822ecb88b6835c12bf4422bef94f19165d896806b66f85eff09dd12603c56: Status 404 returned error can't find the container with id f66822ecb88b6835c12bf4422bef94f19165d896806b66f85eff09dd12603c56
Jul 17 12:27:21.636660 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.636559    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7" event={"ID":"493d99ab-0d82-456d-b003-964aec113cc4","Type":"ContainerStarted","Data":"f66822ecb88b6835c12bf4422bef94f19165d896806b66f85eff09dd12603c56"}
Jul 17 12:27:21.638658 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:21.638597    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd" event={"ID":"fcb3eb8a-314a-44d7-a076-5267528a012c","Type":"ContainerStarted","Data":"2c9c2966bf732827ac44c0c7645a63735c2c958baa28e0864578978ec84e7ffa"}
Jul 17 12:27:22.253832 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.253804    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7866d978f5-b9ml2"]
Jul 17 12:27:22.285240 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.285210    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7866d978f5-b9ml2"]
Jul 17 12:27:22.285380 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.285250    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.295072 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.294504    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 17 12:27:22.392770 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.392713    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-service-ca\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.392909 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.392804    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-console-config\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.392909 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.392829    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-console-oauth-config\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.392909 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.392845    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-oauth-serving-cert\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.392909 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.392866    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jnzq\" (UniqueName: \"kubernetes.io/projected/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-kube-api-access-4jnzq\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.393049 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.392962    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-trusted-ca-bundle\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.393049 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.392993    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-console-serving-cert\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.493675 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.493644    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-service-ca\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.493675 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.493676    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-console-config\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.493905 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.493697    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-console-oauth-config\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.493905 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.493712    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-oauth-serving-cert\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.493905 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.493733    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jnzq\" (UniqueName: \"kubernetes.io/projected/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-kube-api-access-4jnzq\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.493905 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.493796    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-trusted-ca-bundle\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.493905 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.493818    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-console-serving-cert\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.494458 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.494407    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-console-config\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.494559 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.494453    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-service-ca\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.494559 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.494507    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-oauth-serving-cert\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.494682 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.494662    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-trusted-ca-bundle\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.496147 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.496128    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-console-oauth-config\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.496272 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.496252    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-console-serving-cert\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.501377 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.501353    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jnzq\" (UniqueName: \"kubernetes.io/projected/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-kube-api-access-4jnzq\") pod \"console-7866d978f5-b9ml2\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") " pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.601187 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.601136    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:22.803501 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:22.802538    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7866d978f5-b9ml2"]
Jul 17 12:27:22.806700 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:27:22.806663    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601879e4_3c2a_4f8f_ad82_d42740d0a3a0.slice/crio-b3944c23590a4e4eff125e05bebca7a4c07703f2b9d371e40432e0d096396dbe WatchSource:0}: Error finding container b3944c23590a4e4eff125e05bebca7a4c07703f2b9d371e40432e0d096396dbe: Status 404 returned error can't find the container with id b3944c23590a4e4eff125e05bebca7a4c07703f2b9d371e40432e0d096396dbe
Jul 17 12:27:23.647682 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:23.647643    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7866d978f5-b9ml2" event={"ID":"601879e4-3c2a-4f8f-ad82-d42740d0a3a0","Type":"ContainerStarted","Data":"e8a4e2a8e2118fb0e1cb4a52b32f134d270a743777ddf8b12f2ceb87fe59b4e3"}
Jul 17 12:27:23.647967 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:23.647703    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7866d978f5-b9ml2" event={"ID":"601879e4-3c2a-4f8f-ad82-d42740d0a3a0","Type":"ContainerStarted","Data":"b3944c23590a4e4eff125e05bebca7a4c07703f2b9d371e40432e0d096396dbe"}
Jul 17 12:27:23.649330 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:23.649302    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd" event={"ID":"fcb3eb8a-314a-44d7-a076-5267528a012c","Type":"ContainerStarted","Data":"398d91f5a5f276d4d087e0ee13ab8f8c32372843cae6d48672cc9d75c67681cf"}
Jul 17 12:27:23.672551 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:23.672508    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7866d978f5-b9ml2" podStartSLOduration=1.672493944 podStartE2EDuration="1.672493944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:23.671278179 +0000 UTC m=+318.675325075" watchObservedRunningTime="2026-07-17 12:27:23.672493944 +0000 UTC m=+318.676540826"
Jul 17 12:27:23.689477 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:23.689435    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd" podStartSLOduration=1.70843867 podStartE2EDuration="3.689421726s" totalImagesPullingTime="1.980983056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:27:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:23.689362796 +0000 UTC m=+318.693409677" watchObservedRunningTime="2026-07-17 12:27:23.689421726 +0000 UTC m=+318.693468650"
Jul 17 12:27:24.654219 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:24.654190    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7" event={"ID":"493d99ab-0d82-456d-b003-964aec113cc4","Type":"ContainerStarted","Data":"32cf8318b2e15c69c2ab2f9bbd7205de550938d8d947fee4599220c215e280ab"}
Jul 17 12:27:26.663352 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:26.663307    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7" event={"ID":"493d99ab-0d82-456d-b003-964aec113cc4","Type":"ContainerStarted","Data":"1b7fa545cda29a1cbb7891c0a20f31774056f45c6c2732670433394d4b65d1ae"}
Jul 17 12:27:26.663730 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:26.663372    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7" event={"ID":"493d99ab-0d82-456d-b003-964aec113cc4","Type":"ContainerStarted","Data":"018203bbc61e3e758273e885183be691bed3969b32c0bf6936d29355358df8aa"}
Jul 17 12:27:26.688549 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:26.688490    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-587b64d868-wpsz7" podStartSLOduration=1.389673169 podStartE2EDuration="5.688469177s" totalImagesPullingTime="4.298796008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:27:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:26.686533007 +0000 UTC m=+321.690579887" watchObservedRunningTime="2026-07-17 12:27:26.688469177 +0000 UTC m=+321.692516063"
Jul 17 12:27:32.601694 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:32.601665    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:32.602197 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:32.601912    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:32.606419 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:32.606399    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:32.689490 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:32.689458    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:27:40.509773 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:40.509720    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:40.510228 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:40.509811    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:27:43.594559 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:43.594492    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c4665b67d-kc9q2" podUID="9fdd766d-edc4-4aa8-96c9-e0c693d27ce8" containerName="console" containerID="cri-o://433cd6e574e546ba2cc6c831b848467f500607c29e795ea688b860dfa1b13fd7" gracePeriod=15
Jul 17 12:27:43.727960 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:43.727936    2569 generic.go:356] "Generic (PLEG): container finished" podID="9fdd766d-edc4-4aa8-96c9-e0c693d27ce8" containerID="433cd6e574e546ba2cc6c831b848467f500607c29e795ea688b860dfa1b13fd7" exitCode=0
Jul 17 12:27:43.728064 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:43.728008    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4665b67d-kc9q2" event={"ID":"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8","Type":"ContainerDied","Data":"433cd6e574e546ba2cc6c831b848467f500607c29e795ea688b860dfa1b13fd7"}
Jul 17 12:27:43.859154 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:43.859105    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:27:43.960173 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:43.960146    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-console-config\") pod \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") "
Jul 17 12:27:43.960282 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:43.960188    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-service-ca\") pod \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") "
Jul 17 12:27:43.960282 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:43.960256    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-oauth-serving-cert\") pod \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") "
Jul 17 12:27:43.960357 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:43.960320    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-console-serving-cert\") pod \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") "
Jul 17 12:27:43.960357 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:43.960346    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdd8z\" (UniqueName: \"kubernetes.io/projected/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-kube-api-access-vdd8z\") pod \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") "
Jul 17 12:27:43.960450 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:43.960372    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-console-oauth-config\") pod \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\" (UID: \"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8\") "
Jul 17 12:27:43.960609 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:43.960576    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-console-config" (OuterVolumeSpecName: "console-config") pod "9fdd766d-edc4-4aa8-96c9-e0c693d27ce8" (UID: "9fdd766d-edc4-4aa8-96c9-e0c693d27ce8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:27:43.960704 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:43.960659    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9fdd766d-edc4-4aa8-96c9-e0c693d27ce8" (UID: "9fdd766d-edc4-4aa8-96c9-e0c693d27ce8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:27:43.960704 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:43.960687    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-service-ca" (OuterVolumeSpecName: "service-ca") pod "9fdd766d-edc4-4aa8-96c9-e0c693d27ce8" (UID: "9fdd766d-edc4-4aa8-96c9-e0c693d27ce8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:27:43.962396 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:43.962369    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9fdd766d-edc4-4aa8-96c9-e0c693d27ce8" (UID: "9fdd766d-edc4-4aa8-96c9-e0c693d27ce8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:27:43.962483 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:43.962446    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9fdd766d-edc4-4aa8-96c9-e0c693d27ce8" (UID: "9fdd766d-edc4-4aa8-96c9-e0c693d27ce8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:27:43.962573 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:43.962546    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-kube-api-access-vdd8z" (OuterVolumeSpecName: "kube-api-access-vdd8z") pod "9fdd766d-edc4-4aa8-96c9-e0c693d27ce8" (UID: "9fdd766d-edc4-4aa8-96c9-e0c693d27ce8"). InnerVolumeSpecName "kube-api-access-vdd8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:27:44.061117 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:44.061087    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:27:44.061117 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:44.061110    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-oauth-serving-cert\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:27:44.061244 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:44.061124    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-console-serving-cert\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:27:44.061244 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:44.061138    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdd8z\" (UniqueName: \"kubernetes.io/projected/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-kube-api-access-vdd8z\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:27:44.061244 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:44.061150    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-console-oauth-config\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:27:44.061244 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:44.061163    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8-console-config\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:27:44.733499 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:44.733471    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4665b67d-kc9q2"
Jul 17 12:27:44.733499 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:44.733483    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4665b67d-kc9q2" event={"ID":"9fdd766d-edc4-4aa8-96c9-e0c693d27ce8","Type":"ContainerDied","Data":"1081e55899bc36b06fc3f444937251eecc292c35147cb73fcfbefd76832ae80f"}
Jul 17 12:27:44.734012 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:44.733551    2569 scope.go:117] "RemoveContainer" containerID="433cd6e574e546ba2cc6c831b848467f500607c29e795ea688b860dfa1b13fd7"
Jul 17 12:27:44.754576 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:44.754550    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c4665b67d-kc9q2"]
Jul 17 12:27:44.759995 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:44.759967    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c4665b67d-kc9q2"]
Jul 17 12:27:45.632216 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:45.632190    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fdd766d-edc4-4aa8-96c9-e0c693d27ce8" path="/var/lib/kubelet/pods/9fdd766d-edc4-4aa8-96c9-e0c693d27ce8/volumes"
Jul 17 12:27:55.436455 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.436421    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bdfbbb677-g2hsb"]
Jul 17 12:27:55.436944 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.436713    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fdd766d-edc4-4aa8-96c9-e0c693d27ce8" containerName="console"
Jul 17 12:27:55.436944 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.436725    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fdd766d-edc4-4aa8-96c9-e0c693d27ce8" containerName="console"
Jul 17 12:27:55.436944 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.436810    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fdd766d-edc4-4aa8-96c9-e0c693d27ce8" containerName="console"
Jul 17 12:27:55.472513 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.472487    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bdfbbb677-g2hsb"]
Jul 17 12:27:55.472645 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.472570    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.637662 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.637634    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-service-ca\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.637807 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.637670    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-oauth-serving-cert\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.637807 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.637721    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-trusted-ca-bundle\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.637807 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.637741    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dec85784-2d99-4fe9-ac23-556d09751017-console-serving-cert\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.637807 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.637793    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-console-config\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.637962 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.637832    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4ldw\" (UniqueName: \"kubernetes.io/projected/dec85784-2d99-4fe9-ac23-556d09751017-kube-api-access-m4ldw\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.637962 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.637862    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dec85784-2d99-4fe9-ac23-556d09751017-console-oauth-config\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.739030 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.739010    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-trusted-ca-bundle\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.739153 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.739043    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dec85784-2d99-4fe9-ac23-556d09751017-console-serving-cert\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.739153 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.739072    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-console-config\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.739153 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.739112    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4ldw\" (UniqueName: \"kubernetes.io/projected/dec85784-2d99-4fe9-ac23-556d09751017-kube-api-access-m4ldw\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.739465 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.739151    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dec85784-2d99-4fe9-ac23-556d09751017-console-oauth-config\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.739465 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.739294    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-service-ca\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.739465 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.739380    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-oauth-serving-cert\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.740129 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.739850    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-console-config\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.740129 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.739979    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-trusted-ca-bundle\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.740129 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.740068    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-oauth-serving-cert\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.740315 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.740195    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-service-ca\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.742063 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.742046    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dec85784-2d99-4fe9-ac23-556d09751017-console-serving-cert\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.742207 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.742188    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dec85784-2d99-4fe9-ac23-556d09751017-console-oauth-config\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.747020 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.746999    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4ldw\" (UniqueName: \"kubernetes.io/projected/dec85784-2d99-4fe9-ac23-556d09751017-kube-api-access-m4ldw\") pod \"console-5bdfbbb677-g2hsb\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") " pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.780989 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.780965    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:27:55.926372 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:55.926343    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bdfbbb677-g2hsb"]
Jul 17 12:27:55.928921 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:27:55.928892    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec85784_2d99_4fe9_ac23_556d09751017.slice/crio-3cf957fdc52318ecde3735cc37de333a55371846a2abafee770306cbc90b7407 WatchSource:0}: Error finding container 3cf957fdc52318ecde3735cc37de333a55371846a2abafee770306cbc90b7407: Status 404 returned error can't find the container with id 3cf957fdc52318ecde3735cc37de333a55371846a2abafee770306cbc90b7407
Jul 17 12:27:56.778616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:56.778584    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdfbbb677-g2hsb" event={"ID":"dec85784-2d99-4fe9-ac23-556d09751017","Type":"ContainerStarted","Data":"3c4c1d78b7cbd4c9f76a000907cfcfbe4b4e085589b0c73e38ba3af1e924387c"}
Jul 17 12:27:56.778968 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:56.778632    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdfbbb677-g2hsb" event={"ID":"dec85784-2d99-4fe9-ac23-556d09751017","Type":"ContainerStarted","Data":"3cf957fdc52318ecde3735cc37de333a55371846a2abafee770306cbc90b7407"}
Jul 17 12:27:56.796788 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:27:56.796724    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bdfbbb677-g2hsb" podStartSLOduration=1.796709285 podStartE2EDuration="1.796709285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:56.794684324 +0000 UTC m=+351.798731217" watchObservedRunningTime="2026-07-17 12:27:56.796709285 +0000 UTC m=+351.800756167"
Jul 17 12:28:00.515609 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:00.515522    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:28:00.519615 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:00.519586    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-85cbb9955f-4hzpd"
Jul 17 12:28:05.782075 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:05.782040    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:28:05.782530 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:05.782099    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:28:05.786807 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:05.786788    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:28:05.815269 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:05.815247    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:28:05.860915 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:05.860884    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7866d978f5-b9ml2"]
Jul 17 12:28:19.720245 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:19.720206    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs\") pod \"network-metrics-daemon-4nbwg\" (UID: \"966fc2f5-1448-4a14-8b27-21e714628e1a\") " pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:28:19.722535 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:19.722514    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/966fc2f5-1448-4a14-8b27-21e714628e1a-metrics-certs\") pod \"network-metrics-daemon-4nbwg\" (UID: \"966fc2f5-1448-4a14-8b27-21e714628e1a\") " pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:28:19.834135 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:19.834109    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vpz7h\""
Jul 17 12:28:19.841242 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:19.841225    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4nbwg"
Jul 17 12:28:19.967735 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:19.967714    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4nbwg"]
Jul 17 12:28:19.969556 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:28:19.969523    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966fc2f5_1448_4a14_8b27_21e714628e1a.slice/crio-8b855f69fe60737104d6c7c5779a50c8cf596c2526201cc7f54d1e363d94e54a WatchSource:0}: Error finding container 8b855f69fe60737104d6c7c5779a50c8cf596c2526201cc7f54d1e363d94e54a: Status 404 returned error can't find the container with id 8b855f69fe60737104d6c7c5779a50c8cf596c2526201cc7f54d1e363d94e54a
Jul 17 12:28:20.869420 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:20.869392    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4nbwg" event={"ID":"966fc2f5-1448-4a14-8b27-21e714628e1a","Type":"ContainerStarted","Data":"8b855f69fe60737104d6c7c5779a50c8cf596c2526201cc7f54d1e363d94e54a"}
Jul 17 12:28:21.875818 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:21.875771    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4nbwg" event={"ID":"966fc2f5-1448-4a14-8b27-21e714628e1a","Type":"ContainerStarted","Data":"5b4f4240bf33714ffc8d28693d324f792eb6c7e92edcea0377445e4280d381c1"}
Jul 17 12:28:21.876228 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:21.875838    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4nbwg" event={"ID":"966fc2f5-1448-4a14-8b27-21e714628e1a","Type":"ContainerStarted","Data":"2ff32184538c74d2117b2b65d1d0992fe19f9c689e4819651def611da9702323"}
Jul 17 12:28:21.892882 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:21.892828    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4nbwg" podStartSLOduration=375.851514014 podStartE2EDuration="6m16.892811393s" totalImagesPullingTime="1.041297379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:28:21.891589226 +0000 UTC m=+376.895636110" watchObservedRunningTime="2026-07-17 12:28:21.892811393 +0000 UTC m=+376.896858317"
Jul 17 12:28:30.888639 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:30.888564    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7866d978f5-b9ml2" podUID="601879e4-3c2a-4f8f-ad82-d42740d0a3a0" containerName="console" containerID="cri-o://e8a4e2a8e2118fb0e1cb4a52b32f134d270a743777ddf8b12f2ceb87fe59b4e3" gracePeriod=15
Jul 17 12:28:31.131346 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.131324    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:28:31.204733 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.204673    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-trusted-ca-bundle\") pod \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") "
Jul 17 12:28:31.204733 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.204714    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-oauth-serving-cert\") pod \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") "
Jul 17 12:28:31.204911 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.204735    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-console-serving-cert\") pod \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") "
Jul 17 12:28:31.204911 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.204786    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-console-oauth-config\") pod \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") "
Jul 17 12:28:31.204911 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.204828    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-console-config\") pod \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") "
Jul 17 12:28:31.204911 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.204866    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-service-ca\") pod \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") "
Jul 17 12:28:31.205082 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.204920    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jnzq\" (UniqueName: \"kubernetes.io/projected/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-kube-api-access-4jnzq\") pod \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\" (UID: \"601879e4-3c2a-4f8f-ad82-d42740d0a3a0\") "
Jul 17 12:28:31.205245 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.205204    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "601879e4-3c2a-4f8f-ad82-d42740d0a3a0" (UID: "601879e4-3c2a-4f8f-ad82-d42740d0a3a0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:28:31.205245 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.205214    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "601879e4-3c2a-4f8f-ad82-d42740d0a3a0" (UID: "601879e4-3c2a-4f8f-ad82-d42740d0a3a0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:28:31.205423 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.205336    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-service-ca" (OuterVolumeSpecName: "service-ca") pod "601879e4-3c2a-4f8f-ad82-d42740d0a3a0" (UID: "601879e4-3c2a-4f8f-ad82-d42740d0a3a0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:28:31.205423 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.205416    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-console-config" (OuterVolumeSpecName: "console-config") pod "601879e4-3c2a-4f8f-ad82-d42740d0a3a0" (UID: "601879e4-3c2a-4f8f-ad82-d42740d0a3a0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:28:31.206923 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.206898    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "601879e4-3c2a-4f8f-ad82-d42740d0a3a0" (UID: "601879e4-3c2a-4f8f-ad82-d42740d0a3a0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:28:31.207028 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.207010    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-kube-api-access-4jnzq" (OuterVolumeSpecName: "kube-api-access-4jnzq") pod "601879e4-3c2a-4f8f-ad82-d42740d0a3a0" (UID: "601879e4-3c2a-4f8f-ad82-d42740d0a3a0"). InnerVolumeSpecName "kube-api-access-4jnzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:28:31.207076 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.207063    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "601879e4-3c2a-4f8f-ad82-d42740d0a3a0" (UID: "601879e4-3c2a-4f8f-ad82-d42740d0a3a0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:28:31.306259 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.306240    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jnzq\" (UniqueName: \"kubernetes.io/projected/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-kube-api-access-4jnzq\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:28:31.306259 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.306258    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-trusted-ca-bundle\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:28:31.306378 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.306267    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-oauth-serving-cert\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:28:31.306378 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.306275    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-console-serving-cert\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:28:31.306378 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.306284    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-console-oauth-config\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:28:31.306378 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.306292    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-console-config\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:28:31.306378 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.306301    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/601879e4-3c2a-4f8f-ad82-d42740d0a3a0-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:28:31.921367 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.921300    2569 generic.go:356] "Generic (PLEG): container finished" podID="601879e4-3c2a-4f8f-ad82-d42740d0a3a0" containerID="e8a4e2a8e2118fb0e1cb4a52b32f134d270a743777ddf8b12f2ceb87fe59b4e3" exitCode=0
Jul 17 12:28:31.921367 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.921347    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7866d978f5-b9ml2"
Jul 17 12:28:31.921801 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.921370    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7866d978f5-b9ml2" event={"ID":"601879e4-3c2a-4f8f-ad82-d42740d0a3a0","Type":"ContainerDied","Data":"e8a4e2a8e2118fb0e1cb4a52b32f134d270a743777ddf8b12f2ceb87fe59b4e3"}
Jul 17 12:28:31.921801 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.921410    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7866d978f5-b9ml2" event={"ID":"601879e4-3c2a-4f8f-ad82-d42740d0a3a0","Type":"ContainerDied","Data":"b3944c23590a4e4eff125e05bebca7a4c07703f2b9d371e40432e0d096396dbe"}
Jul 17 12:28:31.921801 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.921432    2569 scope.go:117] "RemoveContainer" containerID="e8a4e2a8e2118fb0e1cb4a52b32f134d270a743777ddf8b12f2ceb87fe59b4e3"
Jul 17 12:28:31.930309 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.930290    2569 scope.go:117] "RemoveContainer" containerID="e8a4e2a8e2118fb0e1cb4a52b32f134d270a743777ddf8b12f2ceb87fe59b4e3"
Jul 17 12:28:31.930605 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:28:31.930583    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a4e2a8e2118fb0e1cb4a52b32f134d270a743777ddf8b12f2ceb87fe59b4e3\": container with ID starting with e8a4e2a8e2118fb0e1cb4a52b32f134d270a743777ddf8b12f2ceb87fe59b4e3 not found: ID does not exist" containerID="e8a4e2a8e2118fb0e1cb4a52b32f134d270a743777ddf8b12f2ceb87fe59b4e3"
Jul 17 12:28:31.930660 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.930615    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a4e2a8e2118fb0e1cb4a52b32f134d270a743777ddf8b12f2ceb87fe59b4e3"} err="failed to get container status \"e8a4e2a8e2118fb0e1cb4a52b32f134d270a743777ddf8b12f2ceb87fe59b4e3\": rpc error: code = NotFound desc = could not find container \"e8a4e2a8e2118fb0e1cb4a52b32f134d270a743777ddf8b12f2ceb87fe59b4e3\": container with ID starting with e8a4e2a8e2118fb0e1cb4a52b32f134d270a743777ddf8b12f2ceb87fe59b4e3 not found: ID does not exist"
Jul 17 12:28:31.937624 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.937598    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7866d978f5-b9ml2"]
Jul 17 12:28:31.939699 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:31.939656    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7866d978f5-b9ml2"]
Jul 17 12:28:33.093074 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.093042    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6ddcf4dc7-zcr45"]
Jul 17 12:28:33.093452 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.093339    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="601879e4-3c2a-4f8f-ad82-d42740d0a3a0" containerName="console"
Jul 17 12:28:33.093452 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.093352    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="601879e4-3c2a-4f8f-ad82-d42740d0a3a0" containerName="console"
Jul 17 12:28:33.093452 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.093405    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="601879e4-3c2a-4f8f-ad82-d42740d0a3a0" containerName="console"
Jul 17 12:28:33.096101 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.096084    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.108439 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.108415    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ddcf4dc7-zcr45"]
Jul 17 12:28:33.220666 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.220645    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cdf38e6-ee16-4216-8a1f-209c9a15a811-console-serving-cert\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.220817 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.220676    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-oauth-serving-cert\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.220817 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.220698    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-service-ca\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.220817 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.220713    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-trusted-ca-bundle\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.220817 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.220734    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7hz2\" (UniqueName: \"kubernetes.io/projected/6cdf38e6-ee16-4216-8a1f-209c9a15a811-kube-api-access-c7hz2\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.220986 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.220818    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cdf38e6-ee16-4216-8a1f-209c9a15a811-console-oauth-config\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.220986 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.220837    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-console-config\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.321433 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.321401    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cdf38e6-ee16-4216-8a1f-209c9a15a811-console-oauth-config\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.321433 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.321432    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-console-config\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.321569 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.321461    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cdf38e6-ee16-4216-8a1f-209c9a15a811-console-serving-cert\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.321569 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.321481    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-oauth-serving-cert\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.321569 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.321506    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-service-ca\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.321569 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.321527    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-trusted-ca-bundle\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.321569 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.321551    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7hz2\" (UniqueName: \"kubernetes.io/projected/6cdf38e6-ee16-4216-8a1f-209c9a15a811-kube-api-access-c7hz2\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.322236 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.322213    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-oauth-serving-cert\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.322358 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.322324    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-service-ca\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.322358 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.322342    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-console-config\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.322482 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.322441    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-trusted-ca-bundle\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.323981 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.323963    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cdf38e6-ee16-4216-8a1f-209c9a15a811-console-oauth-config\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.324083 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.324068    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cdf38e6-ee16-4216-8a1f-209c9a15a811-console-serving-cert\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.330273 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.330253    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7hz2\" (UniqueName: \"kubernetes.io/projected/6cdf38e6-ee16-4216-8a1f-209c9a15a811-kube-api-access-c7hz2\") pod \"console-6ddcf4dc7-zcr45\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") " pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.405422 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.405364    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:33.527993 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.527968    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ddcf4dc7-zcr45"]
Jul 17 12:28:33.530347 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:28:33.530321    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cdf38e6_ee16_4216_8a1f_209c9a15a811.slice/crio-d3fae71bb40f8550c1e9c9de197121459f94ffd2984cf49c7c1c4b21a3eb3438 WatchSource:0}: Error finding container d3fae71bb40f8550c1e9c9de197121459f94ffd2984cf49c7c1c4b21a3eb3438: Status 404 returned error can't find the container with id d3fae71bb40f8550c1e9c9de197121459f94ffd2984cf49c7c1c4b21a3eb3438
Jul 17 12:28:33.634791 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.634735    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="601879e4-3c2a-4f8f-ad82-d42740d0a3a0" path="/var/lib/kubelet/pods/601879e4-3c2a-4f8f-ad82-d42740d0a3a0/volumes"
Jul 17 12:28:33.932934 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.932900    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ddcf4dc7-zcr45" event={"ID":"6cdf38e6-ee16-4216-8a1f-209c9a15a811","Type":"ContainerStarted","Data":"cd5ae845bc7206bed804b1e78a846ead4e21758b76778dc655f47120fd95228c"}
Jul 17 12:28:33.933106 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.932947    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ddcf4dc7-zcr45" event={"ID":"6cdf38e6-ee16-4216-8a1f-209c9a15a811","Type":"ContainerStarted","Data":"d3fae71bb40f8550c1e9c9de197121459f94ffd2984cf49c7c1c4b21a3eb3438"}
Jul 17 12:28:33.953346 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:33.953294    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6ddcf4dc7-zcr45" podStartSLOduration=0.953280092 podStartE2EDuration="953.280092ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:28:33.951527529 +0000 UTC m=+388.955574412" watchObservedRunningTime="2026-07-17 12:28:33.953280092 +0000 UTC m=+388.957326974"
Jul 17 12:28:43.406215 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:43.406168    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:43.406699 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:43.406259    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:43.411424 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:43.411397    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:43.974052 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:43.974018    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:28:44.020125 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:28:44.020097    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bdfbbb677-g2hsb"]
Jul 17 12:29:09.041774 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.041690    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bdfbbb677-g2hsb" podUID="dec85784-2d99-4fe9-ac23-556d09751017" containerName="console" containerID="cri-o://3c4c1d78b7cbd4c9f76a000907cfcfbe4b4e085589b0c73e38ba3af1e924387c" gracePeriod=15
Jul 17 12:29:09.278787 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.278765    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:29:09.376034 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.375978    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dec85784-2d99-4fe9-ac23-556d09751017-console-serving-cert\") pod \"dec85784-2d99-4fe9-ac23-556d09751017\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") "
Jul 17 12:29:09.376034 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.376018    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-service-ca\") pod \"dec85784-2d99-4fe9-ac23-556d09751017\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") "
Jul 17 12:29:09.376221 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.376050    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-trusted-ca-bundle\") pod \"dec85784-2d99-4fe9-ac23-556d09751017\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") "
Jul 17 12:29:09.376221 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.376067    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-console-config\") pod \"dec85784-2d99-4fe9-ac23-556d09751017\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") "
Jul 17 12:29:09.376221 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.376087    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4ldw\" (UniqueName: \"kubernetes.io/projected/dec85784-2d99-4fe9-ac23-556d09751017-kube-api-access-m4ldw\") pod \"dec85784-2d99-4fe9-ac23-556d09751017\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") "
Jul 17 12:29:09.376221 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.376165    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-oauth-serving-cert\") pod \"dec85784-2d99-4fe9-ac23-556d09751017\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") "
Jul 17 12:29:09.376221 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.376206    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dec85784-2d99-4fe9-ac23-556d09751017-console-oauth-config\") pod \"dec85784-2d99-4fe9-ac23-556d09751017\" (UID: \"dec85784-2d99-4fe9-ac23-556d09751017\") "
Jul 17 12:29:09.376654 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.376578    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-console-config" (OuterVolumeSpecName: "console-config") pod "dec85784-2d99-4fe9-ac23-556d09751017" (UID: "dec85784-2d99-4fe9-ac23-556d09751017"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:29:09.376654 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.376613    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dec85784-2d99-4fe9-ac23-556d09751017" (UID: "dec85784-2d99-4fe9-ac23-556d09751017"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:29:09.376837 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.376649    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-service-ca" (OuterVolumeSpecName: "service-ca") pod "dec85784-2d99-4fe9-ac23-556d09751017" (UID: "dec85784-2d99-4fe9-ac23-556d09751017"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:29:09.376837 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.376723    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dec85784-2d99-4fe9-ac23-556d09751017" (UID: "dec85784-2d99-4fe9-ac23-556d09751017"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:29:09.378246 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.378222    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec85784-2d99-4fe9-ac23-556d09751017-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dec85784-2d99-4fe9-ac23-556d09751017" (UID: "dec85784-2d99-4fe9-ac23-556d09751017"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:29:09.378410 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.378241    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec85784-2d99-4fe9-ac23-556d09751017-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dec85784-2d99-4fe9-ac23-556d09751017" (UID: "dec85784-2d99-4fe9-ac23-556d09751017"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:29:09.378410 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.378329    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec85784-2d99-4fe9-ac23-556d09751017-kube-api-access-m4ldw" (OuterVolumeSpecName: "kube-api-access-m4ldw") pod "dec85784-2d99-4fe9-ac23-556d09751017" (UID: "dec85784-2d99-4fe9-ac23-556d09751017"). InnerVolumeSpecName "kube-api-access-m4ldw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:29:09.476854 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.476829    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dec85784-2d99-4fe9-ac23-556d09751017-console-oauth-config\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:29:09.476854 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.476849    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dec85784-2d99-4fe9-ac23-556d09751017-console-serving-cert\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:29:09.476984 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.476861    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:29:09.476984 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.476870    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-trusted-ca-bundle\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:29:09.476984 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.476882    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-console-config\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:29:09.476984 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.476892    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4ldw\" (UniqueName: \"kubernetes.io/projected/dec85784-2d99-4fe9-ac23-556d09751017-kube-api-access-m4ldw\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:29:09.476984 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:09.476900    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dec85784-2d99-4fe9-ac23-556d09751017-oauth-serving-cert\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:29:10.068122 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:10.068089    2569 generic.go:356] "Generic (PLEG): container finished" podID="dec85784-2d99-4fe9-ac23-556d09751017" containerID="3c4c1d78b7cbd4c9f76a000907cfcfbe4b4e085589b0c73e38ba3af1e924387c" exitCode=0
Jul 17 12:29:10.068501 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:10.068136    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdfbbb677-g2hsb"
Jul 17 12:29:10.068501 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:10.068167    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdfbbb677-g2hsb" event={"ID":"dec85784-2d99-4fe9-ac23-556d09751017","Type":"ContainerDied","Data":"3c4c1d78b7cbd4c9f76a000907cfcfbe4b4e085589b0c73e38ba3af1e924387c"}
Jul 17 12:29:10.068501 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:10.068208    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdfbbb677-g2hsb" event={"ID":"dec85784-2d99-4fe9-ac23-556d09751017","Type":"ContainerDied","Data":"3cf957fdc52318ecde3735cc37de333a55371846a2abafee770306cbc90b7407"}
Jul 17 12:29:10.068501 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:10.068229    2569 scope.go:117] "RemoveContainer" containerID="3c4c1d78b7cbd4c9f76a000907cfcfbe4b4e085589b0c73e38ba3af1e924387c"
Jul 17 12:29:10.076350 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:10.076334    2569 scope.go:117] "RemoveContainer" containerID="3c4c1d78b7cbd4c9f76a000907cfcfbe4b4e085589b0c73e38ba3af1e924387c"
Jul 17 12:29:10.076610 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:29:10.076590    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c4c1d78b7cbd4c9f76a000907cfcfbe4b4e085589b0c73e38ba3af1e924387c\": container with ID starting with 3c4c1d78b7cbd4c9f76a000907cfcfbe4b4e085589b0c73e38ba3af1e924387c not found: ID does not exist" containerID="3c4c1d78b7cbd4c9f76a000907cfcfbe4b4e085589b0c73e38ba3af1e924387c"
Jul 17 12:29:10.076678 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:10.076622    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c4c1d78b7cbd4c9f76a000907cfcfbe4b4e085589b0c73e38ba3af1e924387c"} err="failed to get container status \"3c4c1d78b7cbd4c9f76a000907cfcfbe4b4e085589b0c73e38ba3af1e924387c\": rpc error: code = NotFound desc = could not find container \"3c4c1d78b7cbd4c9f76a000907cfcfbe4b4e085589b0c73e38ba3af1e924387c\": container with ID starting with 3c4c1d78b7cbd4c9f76a000907cfcfbe4b4e085589b0c73e38ba3af1e924387c not found: ID does not exist"
Jul 17 12:29:10.084224 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:10.084193    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bdfbbb677-g2hsb"]
Jul 17 12:29:10.086100 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:10.086079    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bdfbbb677-g2hsb"]
Jul 17 12:29:11.633322 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:29:11.633284    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dec85784-2d99-4fe9-ac23-556d09751017" path="/var/lib/kubelet/pods/dec85784-2d99-4fe9-ac23-556d09751017/volumes"
Jul 17 12:32:13.198041 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:32:13.198012    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:32:13.199656 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:32:13.199133    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:37:13.221908 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:37:13.221798    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:37:13.227343 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:37:13.223709    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:40:45.121459 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.121422    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7jgz4"]
Jul 17 12:40:45.122008 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.121729    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dec85784-2d99-4fe9-ac23-556d09751017" containerName="console"
Jul 17 12:40:45.122008 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.121741    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dec85784-2d99-4fe9-ac23-556d09751017" containerName="console"
Jul 17 12:40:45.122008 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.121813    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dec85784-2d99-4fe9-ac23-556d09751017" containerName="console"
Jul 17 12:40:45.124734 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.124716    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7jgz4"
Jul 17 12:40:45.127251 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.127229    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 17 12:40:45.128390 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.128365    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 17 12:40:45.128503 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.128365    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hnc42\""
Jul 17 12:40:45.132203 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.132168    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7jgz4"]
Jul 17 12:40:45.202413 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.202386    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8bcca38a-c606-46b7-aab0-bc2de0c013f0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7jgz4\" (UID: \"8bcca38a-c606-46b7-aab0-bc2de0c013f0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7jgz4"
Jul 17 12:40:45.202413 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.202420    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9xbw\" (UniqueName: \"kubernetes.io/projected/8bcca38a-c606-46b7-aab0-bc2de0c013f0-kube-api-access-p9xbw\") pod \"cert-manager-webhook-6486ffd997-7jgz4\" (UID: \"8bcca38a-c606-46b7-aab0-bc2de0c013f0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7jgz4"
Jul 17 12:40:45.303232 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.303200    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8bcca38a-c606-46b7-aab0-bc2de0c013f0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7jgz4\" (UID: \"8bcca38a-c606-46b7-aab0-bc2de0c013f0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7jgz4"
Jul 17 12:40:45.303348 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.303236    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9xbw\" (UniqueName: \"kubernetes.io/projected/8bcca38a-c606-46b7-aab0-bc2de0c013f0-kube-api-access-p9xbw\") pod \"cert-manager-webhook-6486ffd997-7jgz4\" (UID: \"8bcca38a-c606-46b7-aab0-bc2de0c013f0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7jgz4"
Jul 17 12:40:45.318109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.318077    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8bcca38a-c606-46b7-aab0-bc2de0c013f0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7jgz4\" (UID: \"8bcca38a-c606-46b7-aab0-bc2de0c013f0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7jgz4"
Jul 17 12:40:45.318197 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.318132    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9xbw\" (UniqueName: \"kubernetes.io/projected/8bcca38a-c606-46b7-aab0-bc2de0c013f0-kube-api-access-p9xbw\") pod \"cert-manager-webhook-6486ffd997-7jgz4\" (UID: \"8bcca38a-c606-46b7-aab0-bc2de0c013f0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7jgz4"
Jul 17 12:40:45.451611 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.451559    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7jgz4"
Jul 17 12:40:45.572615 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.572587    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7jgz4"]
Jul 17 12:40:45.575923 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:40:45.575895    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bcca38a_c606_46b7_aab0_bc2de0c013f0.slice/crio-339a4d3b000b40c3a0d64588274098a4b32b9b8a385c39a4b1f08f316161e960 WatchSource:0}: Error finding container 339a4d3b000b40c3a0d64588274098a4b32b9b8a385c39a4b1f08f316161e960: Status 404 returned error can't find the container with id 339a4d3b000b40c3a0d64588274098a4b32b9b8a385c39a4b1f08f316161e960
Jul 17 12:40:45.577841 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.577820    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 12:40:45.735675 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:45.735640    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7jgz4" event={"ID":"8bcca38a-c606-46b7-aab0-bc2de0c013f0","Type":"ContainerStarted","Data":"339a4d3b000b40c3a0d64588274098a4b32b9b8a385c39a4b1f08f316161e960"}
Jul 17 12:40:49.752190 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:49.752157    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7jgz4" event={"ID":"8bcca38a-c606-46b7-aab0-bc2de0c013f0","Type":"ContainerStarted","Data":"6538c2c77d6db1a39c9602e704798f62a769dc9918a88e8fb0d1f1df20c4ea80"}
Jul 17 12:40:49.752515 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:49.752229    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7jgz4"
Jul 17 12:40:49.768491 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:49.768449    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-7jgz4" podStartSLOduration=0.747490224 podStartE2EDuration="4.768434797s" totalImagesPullingTime="4.020944573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:40:49.767495648 +0000 UTC m=+1124.771542532" watchObservedRunningTime="2026-07-17 12:40:49.768434797 +0000 UTC m=+1124.772481679"
Jul 17 12:40:53.875204 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:53.875171    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"]
Jul 17 12:40:53.878640 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:53.878623    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:53.880960 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:53.880936    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 17 12:40:53.881371 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:53.881354    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 17 12:40:53.881728 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:53.881709    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 17 12:40:53.882182 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:53.882163    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 17 12:40:53.882256 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:53.882231    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 17 12:40:53.882299 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:53.882251    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 17 12:40:53.882299 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:53.882251    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 17 12:40:53.890603 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:53.890583    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"]
Jul 17 12:40:53.963566 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:53.963541    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ec9e3b-052f-436f-a9c4-40e24de8029a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5d9c784d85-5xxnp\" (UID: \"98ec9e3b-052f-436f-a9c4-40e24de8029a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:53.963674 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:53.963575    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/98ec9e3b-052f-436f-a9c4-40e24de8029a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5d9c784d85-5xxnp\" (UID: \"98ec9e3b-052f-436f-a9c4-40e24de8029a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:53.963674 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:53.963601    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rznkd\" (UniqueName: \"kubernetes.io/projected/98ec9e3b-052f-436f-a9c4-40e24de8029a-kube-api-access-rznkd\") pod \"cluster-proxy-proxy-agent-5d9c784d85-5xxnp\" (UID: \"98ec9e3b-052f-436f-a9c4-40e24de8029a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:53.963789 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:53.963675    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/98ec9e3b-052f-436f-a9c4-40e24de8029a-ca\") pod \"cluster-proxy-proxy-agent-5d9c784d85-5xxnp\" (UID: \"98ec9e3b-052f-436f-a9c4-40e24de8029a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:53.963789 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:53.963701    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/98ec9e3b-052f-436f-a9c4-40e24de8029a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5d9c784d85-5xxnp\" (UID: \"98ec9e3b-052f-436f-a9c4-40e24de8029a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:53.963789 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:53.963724    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/98ec9e3b-052f-436f-a9c4-40e24de8029a-hub\") pod \"cluster-proxy-proxy-agent-5d9c784d85-5xxnp\" (UID: \"98ec9e3b-052f-436f-a9c4-40e24de8029a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:54.064722 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:54.064694    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/98ec9e3b-052f-436f-a9c4-40e24de8029a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5d9c784d85-5xxnp\" (UID: \"98ec9e3b-052f-436f-a9c4-40e24de8029a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:54.064847 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:54.064731    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rznkd\" (UniqueName: \"kubernetes.io/projected/98ec9e3b-052f-436f-a9c4-40e24de8029a-kube-api-access-rznkd\") pod \"cluster-proxy-proxy-agent-5d9c784d85-5xxnp\" (UID: \"98ec9e3b-052f-436f-a9c4-40e24de8029a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:54.064847 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:54.064772    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/98ec9e3b-052f-436f-a9c4-40e24de8029a-ca\") pod \"cluster-proxy-proxy-agent-5d9c784d85-5xxnp\" (UID: \"98ec9e3b-052f-436f-a9c4-40e24de8029a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:54.064964 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:54.064943    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/98ec9e3b-052f-436f-a9c4-40e24de8029a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5d9c784d85-5xxnp\" (UID: \"98ec9e3b-052f-436f-a9c4-40e24de8029a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:54.064999 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:54.064980    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/98ec9e3b-052f-436f-a9c4-40e24de8029a-hub\") pod \"cluster-proxy-proxy-agent-5d9c784d85-5xxnp\" (UID: \"98ec9e3b-052f-436f-a9c4-40e24de8029a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:54.065064 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:54.065048    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ec9e3b-052f-436f-a9c4-40e24de8029a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5d9c784d85-5xxnp\" (UID: \"98ec9e3b-052f-436f-a9c4-40e24de8029a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:54.065691 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:54.065670    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ec9e3b-052f-436f-a9c4-40e24de8029a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5d9c784d85-5xxnp\" (UID: \"98ec9e3b-052f-436f-a9c4-40e24de8029a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:54.067141 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:54.067122    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/98ec9e3b-052f-436f-a9c4-40e24de8029a-ca\") pod \"cluster-proxy-proxy-agent-5d9c784d85-5xxnp\" (UID: \"98ec9e3b-052f-436f-a9c4-40e24de8029a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:54.067233 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:54.067199    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/98ec9e3b-052f-436f-a9c4-40e24de8029a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5d9c784d85-5xxnp\" (UID: \"98ec9e3b-052f-436f-a9c4-40e24de8029a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:54.067587 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:54.067570    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/98ec9e3b-052f-436f-a9c4-40e24de8029a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5d9c784d85-5xxnp\" (UID: \"98ec9e3b-052f-436f-a9c4-40e24de8029a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:54.067647 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:54.067632    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/98ec9e3b-052f-436f-a9c4-40e24de8029a-hub\") pod \"cluster-proxy-proxy-agent-5d9c784d85-5xxnp\" (UID: \"98ec9e3b-052f-436f-a9c4-40e24de8029a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:54.072603 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:54.072572    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rznkd\" (UniqueName: \"kubernetes.io/projected/98ec9e3b-052f-436f-a9c4-40e24de8029a-kube-api-access-rznkd\") pod \"cluster-proxy-proxy-agent-5d9c784d85-5xxnp\" (UID: \"98ec9e3b-052f-436f-a9c4-40e24de8029a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:54.187731 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:54.187681    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"
Jul 17 12:40:54.317138 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:54.317112    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp"]
Jul 17 12:40:54.319128 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:40:54.319097    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98ec9e3b_052f_436f_a9c4_40e24de8029a.slice/crio-97fbbbfa81bf2445f5aac7cf24777c6e3d94248324e01b4f6571255014eb74f5 WatchSource:0}: Error finding container 97fbbbfa81bf2445f5aac7cf24777c6e3d94248324e01b4f6571255014eb74f5: Status 404 returned error can't find the container with id 97fbbbfa81bf2445f5aac7cf24777c6e3d94248324e01b4f6571255014eb74f5
Jul 17 12:40:54.772334 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:54.772292    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp" event={"ID":"98ec9e3b-052f-436f-a9c4-40e24de8029a","Type":"ContainerStarted","Data":"97fbbbfa81bf2445f5aac7cf24777c6e3d94248324e01b4f6571255014eb74f5"}
Jul 17 12:40:55.759407 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:40:55.759373    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7jgz4"
Jul 17 12:41:04.839989 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:04.839947    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-h7pq2"]
Jul 17 12:41:04.843135 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:04.843114    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-h7pq2"
Jul 17 12:41:04.845658 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:04.845617    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-zn9g2\""
Jul 17 12:41:04.852558 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:04.852524    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-h7pq2"]
Jul 17 12:41:04.940497 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:04.940472    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c79de4ac-4ac7-40ee-b271-4241b78907da-bound-sa-token\") pod \"cert-manager-55b4d954d5-h7pq2\" (UID: \"c79de4ac-4ac7-40ee-b271-4241b78907da\") " pod="cert-manager/cert-manager-55b4d954d5-h7pq2"
Jul 17 12:41:04.940605 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:04.940509    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd4lk\" (UniqueName: \"kubernetes.io/projected/c79de4ac-4ac7-40ee-b271-4241b78907da-kube-api-access-rd4lk\") pod \"cert-manager-55b4d954d5-h7pq2\" (UID: \"c79de4ac-4ac7-40ee-b271-4241b78907da\") " pod="cert-manager/cert-manager-55b4d954d5-h7pq2"
Jul 17 12:41:05.041761 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:05.041734    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c79de4ac-4ac7-40ee-b271-4241b78907da-bound-sa-token\") pod \"cert-manager-55b4d954d5-h7pq2\" (UID: \"c79de4ac-4ac7-40ee-b271-4241b78907da\") " pod="cert-manager/cert-manager-55b4d954d5-h7pq2"
Jul 17 12:41:05.041866 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:05.041795    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd4lk\" (UniqueName: \"kubernetes.io/projected/c79de4ac-4ac7-40ee-b271-4241b78907da-kube-api-access-rd4lk\") pod \"cert-manager-55b4d954d5-h7pq2\" (UID: \"c79de4ac-4ac7-40ee-b271-4241b78907da\") " pod="cert-manager/cert-manager-55b4d954d5-h7pq2"
Jul 17 12:41:05.049924 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:05.049899    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c79de4ac-4ac7-40ee-b271-4241b78907da-bound-sa-token\") pod \"cert-manager-55b4d954d5-h7pq2\" (UID: \"c79de4ac-4ac7-40ee-b271-4241b78907da\") " pod="cert-manager/cert-manager-55b4d954d5-h7pq2"
Jul 17 12:41:05.050224 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:05.050200    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd4lk\" (UniqueName: \"kubernetes.io/projected/c79de4ac-4ac7-40ee-b271-4241b78907da-kube-api-access-rd4lk\") pod \"cert-manager-55b4d954d5-h7pq2\" (UID: \"c79de4ac-4ac7-40ee-b271-4241b78907da\") " pod="cert-manager/cert-manager-55b4d954d5-h7pq2"
Jul 17 12:41:05.154285 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:05.154225    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-h7pq2"
Jul 17 12:41:05.277953 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:05.277932    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-h7pq2"]
Jul 17 12:41:05.280257 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:41:05.280218    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc79de4ac_4ac7_40ee_b271_4241b78907da.slice/crio-b3c3eb04729e8080953ba7509e8dfe73b2e776d1fabdca16e63b050a864b959c WatchSource:0}: Error finding container b3c3eb04729e8080953ba7509e8dfe73b2e776d1fabdca16e63b050a864b959c: Status 404 returned error can't find the container with id b3c3eb04729e8080953ba7509e8dfe73b2e776d1fabdca16e63b050a864b959c
Jul 17 12:41:05.815544 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:05.815506    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-h7pq2" event={"ID":"c79de4ac-4ac7-40ee-b271-4241b78907da","Type":"ContainerStarted","Data":"2a17f8ce6e1c6a84edc30fb9a36b6c5708901ea9bb672371c3150ffe06fc99b0"}
Jul 17 12:41:05.815697 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:05.815555    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-h7pq2" event={"ID":"c79de4ac-4ac7-40ee-b271-4241b78907da","Type":"ContainerStarted","Data":"b3c3eb04729e8080953ba7509e8dfe73b2e776d1fabdca16e63b050a864b959c"}
Jul 17 12:41:05.833854 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:05.833814    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-h7pq2" podStartSLOduration=1.833800675 podStartE2EDuration="1.833800675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:41:05.833133484 +0000 UTC m=+1140.837180364" watchObservedRunningTime="2026-07-17 12:41:05.833800675 +0000 UTC m=+1140.837847556"
Jul 17 12:41:13.853219 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:13.853178    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp" event={"ID":"98ec9e3b-052f-436f-a9c4-40e24de8029a","Type":"ContainerStarted","Data":"a10368a1310df82fae24791f5ef3e046a01acf7d288225532414d82a036f3c8e"}
Jul 17 12:41:13.853219 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:13.853223    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp" event={"ID":"98ec9e3b-052f-436f-a9c4-40e24de8029a","Type":"ContainerStarted","Data":"382cf19b04d13d86ea04e0b9f16149be77c41406032bd8e738d9ea9d98967694"}
Jul 17 12:41:13.853572 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:13.853235    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp" event={"ID":"98ec9e3b-052f-436f-a9c4-40e24de8029a","Type":"ContainerStarted","Data":"3828f284176f2adc88e3f73d09174978b569146e117299de8703348ba26107b4"}
Jul 17 12:41:13.872884 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:13.872843    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5d9c784d85-5xxnp" podStartSLOduration=1.5370145979999998 podStartE2EDuration="20.872828212s" totalImagesPullingTime="19.335813614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:41:13.870929793 +0000 UTC m=+1148.874976682" watchObservedRunningTime="2026-07-17 12:41:13.872828212 +0000 UTC m=+1148.876875075"
Jul 17 12:41:24.245769 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.245717    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"]
Jul 17 12:41:24.248007 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.247992    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:24.250975 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.250951    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 12:41:24.251138 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.250951    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 12:41:24.251138 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.250949    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p95ns\""
Jul 17 12:41:24.251138 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.251094    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 17 12:41:24.251373 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.251358    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 12:41:24.263691 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.263670    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"]
Jul 17 12:41:24.385457 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.385429    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1272b7db-2ffc-4f97-b60a-693a9b5749d6-manifests\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-ccn8z\" (UID: \"1272b7db-2ffc-4f97-b60a-693a9b5749d6\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:24.385549 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.385460    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1272b7db-2ffc-4f97-b60a-693a9b5749d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-ccn8z\" (UID: \"1272b7db-2ffc-4f97-b60a-693a9b5749d6\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:24.385549 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.385483    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cm6j\" (UniqueName: \"kubernetes.io/projected/1272b7db-2ffc-4f97-b60a-693a9b5749d6-kube-api-access-2cm6j\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-ccn8z\" (UID: \"1272b7db-2ffc-4f97-b60a-693a9b5749d6\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:24.385619 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.385559    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1272b7db-2ffc-4f97-b60a-693a9b5749d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-ccn8z\" (UID: \"1272b7db-2ffc-4f97-b60a-693a9b5749d6\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:24.385619 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.385604    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1272b7db-2ffc-4f97-b60a-693a9b5749d6-tmp\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-ccn8z\" (UID: \"1272b7db-2ffc-4f97-b60a-693a9b5749d6\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:24.486845 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.486815    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1272b7db-2ffc-4f97-b60a-693a9b5749d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-ccn8z\" (UID: \"1272b7db-2ffc-4f97-b60a-693a9b5749d6\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:24.486933 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.486859    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1272b7db-2ffc-4f97-b60a-693a9b5749d6-tmp\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-ccn8z\" (UID: \"1272b7db-2ffc-4f97-b60a-693a9b5749d6\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:24.486933 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.486880    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1272b7db-2ffc-4f97-b60a-693a9b5749d6-manifests\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-ccn8z\" (UID: \"1272b7db-2ffc-4f97-b60a-693a9b5749d6\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:24.486933 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.486901    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1272b7db-2ffc-4f97-b60a-693a9b5749d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-ccn8z\" (UID: \"1272b7db-2ffc-4f97-b60a-693a9b5749d6\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:24.487043 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.486939    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cm6j\" (UniqueName: \"kubernetes.io/projected/1272b7db-2ffc-4f97-b60a-693a9b5749d6-kube-api-access-2cm6j\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-ccn8z\" (UID: \"1272b7db-2ffc-4f97-b60a-693a9b5749d6\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:24.487345 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.487327    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1272b7db-2ffc-4f97-b60a-693a9b5749d6-manifests\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-ccn8z\" (UID: \"1272b7db-2ffc-4f97-b60a-693a9b5749d6\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:24.489297 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.489270    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1272b7db-2ffc-4f97-b60a-693a9b5749d6-tmp\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-ccn8z\" (UID: \"1272b7db-2ffc-4f97-b60a-693a9b5749d6\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:24.489569 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.489547    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1272b7db-2ffc-4f97-b60a-693a9b5749d6-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-ccn8z\" (UID: \"1272b7db-2ffc-4f97-b60a-693a9b5749d6\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:24.489645 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.489622    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1272b7db-2ffc-4f97-b60a-693a9b5749d6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-ccn8z\" (UID: \"1272b7db-2ffc-4f97-b60a-693a9b5749d6\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:24.501368 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.501311    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cm6j\" (UniqueName: \"kubernetes.io/projected/1272b7db-2ffc-4f97-b60a-693a9b5749d6-kube-api-access-2cm6j\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-ccn8z\" (UID: \"1272b7db-2ffc-4f97-b60a-693a9b5749d6\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:24.558268 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.558246    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:24.686535 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.686507    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"]
Jul 17 12:41:24.689008 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:41:24.688982    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1272b7db_2ffc_4f97_b60a_693a9b5749d6.slice/crio-678b03ad3f82c39d621351b08b539b08863fcd9e211b253476ddb983ead1dc8a WatchSource:0}: Error finding container 678b03ad3f82c39d621351b08b539b08863fcd9e211b253476ddb983ead1dc8a: Status 404 returned error can't find the container with id 678b03ad3f82c39d621351b08b539b08863fcd9e211b253476ddb983ead1dc8a
Jul 17 12:41:24.903939 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:24.903868    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z" event={"ID":"1272b7db-2ffc-4f97-b60a-693a9b5749d6","Type":"ContainerStarted","Data":"678b03ad3f82c39d621351b08b539b08863fcd9e211b253476ddb983ead1dc8a"}
Jul 17 12:41:27.922550 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:27.922516    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z" event={"ID":"1272b7db-2ffc-4f97-b60a-693a9b5749d6","Type":"ContainerStarted","Data":"ee40979667bb3425732d0ad4541d114a4cf891509133d4684009d9dc8e7c93a5"}
Jul 17 12:41:28.933399 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:28.933365    2569 generic.go:356] "Generic (PLEG): container finished" podID="1272b7db-2ffc-4f97-b60a-693a9b5749d6" containerID="ee40979667bb3425732d0ad4541d114a4cf891509133d4684009d9dc8e7c93a5" exitCode=0
Jul 17 12:41:28.933818 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:28.933406    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z" event={"ID":"1272b7db-2ffc-4f97-b60a-693a9b5749d6","Type":"ContainerDied","Data":"ee40979667bb3425732d0ad4541d114a4cf891509133d4684009d9dc8e7c93a5"}
Jul 17 12:41:29.940379 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:29.940346    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z" event={"ID":"1272b7db-2ffc-4f97-b60a-693a9b5749d6","Type":"ContainerStarted","Data":"7d7c15593beb2c07928a62537039608dfd1b99b3d3e729d3d9fc518f7c0d5e23"}
Jul 17 12:41:29.940882 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:29.940494    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:29.974481 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:29.974440    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z" podStartSLOduration=1.481117667 podStartE2EDuration="5.974426682s" totalImagesPullingTime="3.485542847s" totalInitContainerRuntime="1.007766168s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:41:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:41:29.973048307 +0000 UTC m=+1164.977095188" watchObservedRunningTime="2026-07-17 12:41:29.974426682 +0000 UTC m=+1164.978473563"
Jul 17 12:41:40.947820 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:40.947786    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-ccn8z"
Jul 17 12:41:51.714221 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:51.714187    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"]
Jul 17 12:41:51.726667 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:51.726641    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"
Jul 17 12:41:51.727536 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:51.727506    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"]
Jul 17 12:41:51.729581 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:51.729560    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 17 12:41:51.729788 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:51.729648    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 17 12:41:51.730842 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:51.730815    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 17 12:41:51.730969 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:51.730872    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 17 12:41:51.731046 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:51.730977    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 17 12:41:51.731046 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:51.730977    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8sx92\""
Jul 17 12:41:51.907426 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:51.907397    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1246e75c-64df-4548-9c61-4d8578615b17-manager-config\") pod \"lws-controller-manager-774b4c8dcd-v4v2x\" (UID: \"1246e75c-64df-4548-9c61-4d8578615b17\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"
Jul 17 12:41:51.907574 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:51.907428    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjb9m\" (UniqueName: \"kubernetes.io/projected/1246e75c-64df-4548-9c61-4d8578615b17-kube-api-access-cjb9m\") pod \"lws-controller-manager-774b4c8dcd-v4v2x\" (UID: \"1246e75c-64df-4548-9c61-4d8578615b17\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"
Jul 17 12:41:51.907574 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:51.907449    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1246e75c-64df-4548-9c61-4d8578615b17-cert\") pod \"lws-controller-manager-774b4c8dcd-v4v2x\" (UID: \"1246e75c-64df-4548-9c61-4d8578615b17\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"
Jul 17 12:41:51.907574 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:51.907489    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1246e75c-64df-4548-9c61-4d8578615b17-metrics-cert\") pod \"lws-controller-manager-774b4c8dcd-v4v2x\" (UID: \"1246e75c-64df-4548-9c61-4d8578615b17\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"
Jul 17 12:41:52.008701 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.008674    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1246e75c-64df-4548-9c61-4d8578615b17-manager-config\") pod \"lws-controller-manager-774b4c8dcd-v4v2x\" (UID: \"1246e75c-64df-4548-9c61-4d8578615b17\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"
Jul 17 12:41:52.008845 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.008709    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjb9m\" (UniqueName: \"kubernetes.io/projected/1246e75c-64df-4548-9c61-4d8578615b17-kube-api-access-cjb9m\") pod \"lws-controller-manager-774b4c8dcd-v4v2x\" (UID: \"1246e75c-64df-4548-9c61-4d8578615b17\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"
Jul 17 12:41:52.008845 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.008729    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1246e75c-64df-4548-9c61-4d8578615b17-cert\") pod \"lws-controller-manager-774b4c8dcd-v4v2x\" (UID: \"1246e75c-64df-4548-9c61-4d8578615b17\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"
Jul 17 12:41:52.008845 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.008744    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1246e75c-64df-4548-9c61-4d8578615b17-metrics-cert\") pod \"lws-controller-manager-774b4c8dcd-v4v2x\" (UID: \"1246e75c-64df-4548-9c61-4d8578615b17\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"
Jul 17 12:41:52.009363 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.009341    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1246e75c-64df-4548-9c61-4d8578615b17-manager-config\") pod \"lws-controller-manager-774b4c8dcd-v4v2x\" (UID: \"1246e75c-64df-4548-9c61-4d8578615b17\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"
Jul 17 12:41:52.011198 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.011173    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1246e75c-64df-4548-9c61-4d8578615b17-metrics-cert\") pod \"lws-controller-manager-774b4c8dcd-v4v2x\" (UID: \"1246e75c-64df-4548-9c61-4d8578615b17\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"
Jul 17 12:41:52.011388 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.011367    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1246e75c-64df-4548-9c61-4d8578615b17-cert\") pod \"lws-controller-manager-774b4c8dcd-v4v2x\" (UID: \"1246e75c-64df-4548-9c61-4d8578615b17\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"
Jul 17 12:41:52.021368 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.021346    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjb9m\" (UniqueName: \"kubernetes.io/projected/1246e75c-64df-4548-9c61-4d8578615b17-kube-api-access-cjb9m\") pod \"lws-controller-manager-774b4c8dcd-v4v2x\" (UID: \"1246e75c-64df-4548-9c61-4d8578615b17\") " pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"
Jul 17 12:41:52.038686 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.038666    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"
Jul 17 12:41:52.177798 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.177769    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"]
Jul 17 12:41:52.179943 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:41:52.179915    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1246e75c_64df_4548_9c61_4d8578615b17.slice/crio-8e9891f8382ba471f11162ff1dc37b9ee333ca9833f419e1e7898af9a893caab WatchSource:0}: Error finding container 8e9891f8382ba471f11162ff1dc37b9ee333ca9833f419e1e7898af9a893caab: Status 404 returned error can't find the container with id 8e9891f8382ba471f11162ff1dc37b9ee333ca9833f419e1e7898af9a893caab
Jul 17 12:41:52.604567 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.604534    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"]
Jul 17 12:41:52.607615 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.607589    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.610527 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.610500    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 17 12:41:52.610949 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.610930    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 12:41:52.611222 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.611203    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-54nr7\""
Jul 17 12:41:52.611315 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.611227    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 12:41:52.611315 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.611259    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 17 12:41:52.611315 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.611300    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 17 12:41:52.611544 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.611526    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 12:41:52.622513 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.622490    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"]
Jul 17 12:41:52.716593 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.716561    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.717012 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.716633    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.717012 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.716723    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.717012 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.716777    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.717012 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.716875    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.717012 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.716910    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jjjv\" (UniqueName: \"kubernetes.io/projected/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-kube-api-access-2jjjv\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.717012 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.716953    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.818324 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.818288    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.818469 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.818364    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.818469 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.818396    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.818469 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.818417    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.818469 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.818453    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.818670 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.818478    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jjjv\" (UniqueName: \"kubernetes.io/projected/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-kube-api-access-2jjjv\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.818670 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.818508    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.819147 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.819117    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.821368 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.821341    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.821670 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.821650    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.821853 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.821829    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.821971 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.821919    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.829012 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.828949    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jjjv\" (UniqueName: \"kubernetes.io/projected/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-kube-api-access-2jjjv\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.829290 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.829247    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-wh79n\" (UID: \"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:52.922744 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:52.922673    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:53.037436 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:53.037343    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x" event={"ID":"1246e75c-64df-4548-9c61-4d8578615b17","Type":"ContainerStarted","Data":"8e9891f8382ba471f11162ff1dc37b9ee333ca9833f419e1e7898af9a893caab"}
Jul 17 12:41:53.117477 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:53.117411    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"]
Jul 17 12:41:53.120440 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:41:53.120310    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ded48e5_a3fc_4fd4_b680_dfd7b27b27bb.slice/crio-d5ef4740511215ff372093435b5b3b16d7459dbeb81a6f13a6faffcae743e2f0 WatchSource:0}: Error finding container d5ef4740511215ff372093435b5b3b16d7459dbeb81a6f13a6faffcae743e2f0: Status 404 returned error can't find the container with id d5ef4740511215ff372093435b5b3b16d7459dbeb81a6f13a6faffcae743e2f0
Jul 17 12:41:54.043912 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:54.043870    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n" event={"ID":"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb","Type":"ContainerStarted","Data":"d5ef4740511215ff372093435b5b3b16d7459dbeb81a6f13a6faffcae743e2f0"}
Jul 17 12:41:55.051599 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:55.051561    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x" event={"ID":"1246e75c-64df-4548-9c61-4d8578615b17","Type":"ContainerStarted","Data":"100092fbf6b2c124c732fe6441f1e183592fe579612b3da633f64355e321bf82"}
Jul 17 12:41:55.052060 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:55.051674    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"
Jul 17 12:41:55.093180 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:55.093129    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x" podStartSLOduration=1.9612091390000002 podStartE2EDuration="4.093109137s" totalImagesPullingTime="2.131899998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:41:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:41:55.090294149 +0000 UTC m=+1190.094341032" watchObservedRunningTime="2026-07-17 12:41:55.093109137 +0000 UTC m=+1190.097156020"
Jul 17 12:41:56.103134 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:56.103095    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 12:41:56.103437 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:56.103158    2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 12:41:57.061840 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:57.061805    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n" event={"ID":"4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb","Type":"ContainerStarted","Data":"08e13c74ab91c77fef5cc1f7456a83a731ef139f2ca36bbaf5d619a9c707dfa5"}
Jul 17 12:41:57.062015 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:57.061976    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:57.063578 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:57.063559    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n"
Jul 17 12:41:57.087131 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:41:57.087082    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wh79n" podStartSLOduration=2.10695906 podStartE2EDuration="5.087068422s" totalImagesPullingTime="2.980109362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:41:57.084925746 +0000 UTC m=+1192.088972630" watchObservedRunningTime="2026-07-17 12:41:57.087068422 +0000 UTC m=+1192.091115304"
Jul 17 12:42:06.058109 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:06.058079    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-774b4c8dcd-v4v2x"
Jul 17 12:42:13.250551 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:13.250516    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:42:13.252953 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:13.251176    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:42:54.226731 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.226693    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b5cc587cc-j84vt"]
Jul 17 12:42:54.232429 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.230885    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.243686 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.243659    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b5cc587cc-j84vt"]
Jul 17 12:42:54.379856 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.379830    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c03aaeb-0c17-4472-8a65-6b168083cce1-trusted-ca-bundle\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.379985 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.379872    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c03aaeb-0c17-4472-8a65-6b168083cce1-service-ca\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.379985 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.379891    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c03aaeb-0c17-4472-8a65-6b168083cce1-console-oauth-config\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.379985 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.379953    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c03aaeb-0c17-4472-8a65-6b168083cce1-oauth-serving-cert\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.379985 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.379976    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c03aaeb-0c17-4472-8a65-6b168083cce1-console-config\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.380118 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.380004    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqms4\" (UniqueName: \"kubernetes.io/projected/8c03aaeb-0c17-4472-8a65-6b168083cce1-kube-api-access-lqms4\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.380118 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.380023    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c03aaeb-0c17-4472-8a65-6b168083cce1-console-serving-cert\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.480820 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.480736    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c03aaeb-0c17-4472-8a65-6b168083cce1-console-oauth-config\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.480820 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.480811    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c03aaeb-0c17-4472-8a65-6b168083cce1-oauth-serving-cert\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.481020 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.480844    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c03aaeb-0c17-4472-8a65-6b168083cce1-console-config\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.481020 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.480912    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqms4\" (UniqueName: \"kubernetes.io/projected/8c03aaeb-0c17-4472-8a65-6b168083cce1-kube-api-access-lqms4\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.481020 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.480948    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c03aaeb-0c17-4472-8a65-6b168083cce1-console-serving-cert\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.481020 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.480992    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c03aaeb-0c17-4472-8a65-6b168083cce1-trusted-ca-bundle\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.481191 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.481029    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c03aaeb-0c17-4472-8a65-6b168083cce1-service-ca\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.481664 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.481630    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c03aaeb-0c17-4472-8a65-6b168083cce1-oauth-serving-cert\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.481795 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.481744    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c03aaeb-0c17-4472-8a65-6b168083cce1-console-config\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.481849 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.481801    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c03aaeb-0c17-4472-8a65-6b168083cce1-service-ca\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.482181 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.482151    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c03aaeb-0c17-4472-8a65-6b168083cce1-trusted-ca-bundle\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.483347 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.483311    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c03aaeb-0c17-4472-8a65-6b168083cce1-console-oauth-config\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.483642 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.483623    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c03aaeb-0c17-4472-8a65-6b168083cce1-console-serving-cert\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.489924 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.489898    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqms4\" (UniqueName: \"kubernetes.io/projected/8c03aaeb-0c17-4472-8a65-6b168083cce1-kube-api-access-lqms4\") pod \"console-7b5cc587cc-j84vt\" (UID: \"8c03aaeb-0c17-4472-8a65-6b168083cce1\") " pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.543723 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.543698    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:42:54.673353 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:54.673327    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b5cc587cc-j84vt"]
Jul 17 12:42:54.676087 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:42:54.676035    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c03aaeb_0c17_4472_8a65_6b168083cce1.slice/crio-a8c64a0f9c76a5230ac200610193050d4763b072b432adcc59347b8e93b77287 WatchSource:0}: Error finding container a8c64a0f9c76a5230ac200610193050d4763b072b432adcc59347b8e93b77287: Status 404 returned error can't find the container with id a8c64a0f9c76a5230ac200610193050d4763b072b432adcc59347b8e93b77287
Jul 17 12:42:55.316144 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:55.316097    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b5cc587cc-j84vt" event={"ID":"8c03aaeb-0c17-4472-8a65-6b168083cce1","Type":"ContainerStarted","Data":"5454663aa07c3a825613a220367b267975aa81663c2b6f558a08999cbf5ac563"}
Jul 17 12:42:55.316573 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:55.316167    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b5cc587cc-j84vt" event={"ID":"8c03aaeb-0c17-4472-8a65-6b168083cce1","Type":"ContainerStarted","Data":"a8c64a0f9c76a5230ac200610193050d4763b072b432adcc59347b8e93b77287"}
Jul 17 12:42:55.334827 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:42:55.334770    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b5cc587cc-j84vt" podStartSLOduration=1.334739573 podStartE2EDuration="1.334739573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:42:55.332772076 +0000 UTC m=+1250.336818959" watchObservedRunningTime="2026-07-17 12:42:55.334739573 +0000 UTC m=+1250.338786455"
Jul 17 12:43:04.544897 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:04.544800    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:43:04.544897 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:04.544877    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:43:04.549870 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:04.549841    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:43:05.364870 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:05.364825    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b5cc587cc-j84vt"
Jul 17 12:43:05.409825 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:05.409794    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ddcf4dc7-zcr45"]
Jul 17 12:43:30.430901 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.430840    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6ddcf4dc7-zcr45" podUID="6cdf38e6-ee16-4216-8a1f-209c9a15a811" containerName="console" containerID="cri-o://cd5ae845bc7206bed804b1e78a846ead4e21758b76778dc655f47120fd95228c" gracePeriod=15
Jul 17 12:43:30.675625 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.675605    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:43:30.846330 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.846302    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-oauth-serving-cert\") pod \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") "
Jul 17 12:43:30.846466 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.846362    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7hz2\" (UniqueName: \"kubernetes.io/projected/6cdf38e6-ee16-4216-8a1f-209c9a15a811-kube-api-access-c7hz2\") pod \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") "
Jul 17 12:43:30.846466 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.846398    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-console-config\") pod \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") "
Jul 17 12:43:30.846466 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.846419    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-trusted-ca-bundle\") pod \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") "
Jul 17 12:43:30.846466 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.846445    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cdf38e6-ee16-4216-8a1f-209c9a15a811-console-serving-cert\") pod \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") "
Jul 17 12:43:30.846674 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.846472    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cdf38e6-ee16-4216-8a1f-209c9a15a811-console-oauth-config\") pod \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") "
Jul 17 12:43:30.846674 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.846525    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-service-ca\") pod \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\" (UID: \"6cdf38e6-ee16-4216-8a1f-209c9a15a811\") "
Jul 17 12:43:30.847328 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.847293    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6cdf38e6-ee16-4216-8a1f-209c9a15a811" (UID: "6cdf38e6-ee16-4216-8a1f-209c9a15a811"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:43:30.847531 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.847338    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-console-config" (OuterVolumeSpecName: "console-config") pod "6cdf38e6-ee16-4216-8a1f-209c9a15a811" (UID: "6cdf38e6-ee16-4216-8a1f-209c9a15a811"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:43:30.847531 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.847383    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6cdf38e6-ee16-4216-8a1f-209c9a15a811" (UID: "6cdf38e6-ee16-4216-8a1f-209c9a15a811"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:43:30.847919 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.847792    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-service-ca" (OuterVolumeSpecName: "service-ca") pod "6cdf38e6-ee16-4216-8a1f-209c9a15a811" (UID: "6cdf38e6-ee16-4216-8a1f-209c9a15a811"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:43:30.848608 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.848578    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cdf38e6-ee16-4216-8a1f-209c9a15a811-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6cdf38e6-ee16-4216-8a1f-209c9a15a811" (UID: "6cdf38e6-ee16-4216-8a1f-209c9a15a811"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:43:30.848702 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.848640    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cdf38e6-ee16-4216-8a1f-209c9a15a811-kube-api-access-c7hz2" (OuterVolumeSpecName: "kube-api-access-c7hz2") pod "6cdf38e6-ee16-4216-8a1f-209c9a15a811" (UID: "6cdf38e6-ee16-4216-8a1f-209c9a15a811"). InnerVolumeSpecName "kube-api-access-c7hz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:43:30.848774 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.848721    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cdf38e6-ee16-4216-8a1f-209c9a15a811-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6cdf38e6-ee16-4216-8a1f-209c9a15a811" (UID: "6cdf38e6-ee16-4216-8a1f-209c9a15a811"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:43:30.948352 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.948332    2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-oauth-serving-cert\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:43:30.948352 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.948352    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7hz2\" (UniqueName: \"kubernetes.io/projected/6cdf38e6-ee16-4216-8a1f-209c9a15a811-kube-api-access-c7hz2\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:43:30.948481 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.948362    2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-console-config\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:43:30.948481 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.948371    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-trusted-ca-bundle\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:43:30.948481 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.948379    2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cdf38e6-ee16-4216-8a1f-209c9a15a811-console-serving-cert\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:43:30.948481 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.948388    2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cdf38e6-ee16-4216-8a1f-209c9a15a811-console-oauth-config\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:43:30.948481 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:30.948396    2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cdf38e6-ee16-4216-8a1f-209c9a15a811-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 12:43:31.486126 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:31.486087    2569 generic.go:356] "Generic (PLEG): container finished" podID="6cdf38e6-ee16-4216-8a1f-209c9a15a811" containerID="cd5ae845bc7206bed804b1e78a846ead4e21758b76778dc655f47120fd95228c" exitCode=0
Jul 17 12:43:31.486528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:31.486140    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ddcf4dc7-zcr45"
Jul 17 12:43:31.486528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:31.486167    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ddcf4dc7-zcr45" event={"ID":"6cdf38e6-ee16-4216-8a1f-209c9a15a811","Type":"ContainerDied","Data":"cd5ae845bc7206bed804b1e78a846ead4e21758b76778dc655f47120fd95228c"}
Jul 17 12:43:31.486528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:31.486253    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ddcf4dc7-zcr45" event={"ID":"6cdf38e6-ee16-4216-8a1f-209c9a15a811","Type":"ContainerDied","Data":"d3fae71bb40f8550c1e9c9de197121459f94ffd2984cf49c7c1c4b21a3eb3438"}
Jul 17 12:43:31.486528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:31.486278    2569 scope.go:117] "RemoveContainer" containerID="cd5ae845bc7206bed804b1e78a846ead4e21758b76778dc655f47120fd95228c"
Jul 17 12:43:31.496336 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:31.496317    2569 scope.go:117] "RemoveContainer" containerID="cd5ae845bc7206bed804b1e78a846ead4e21758b76778dc655f47120fd95228c"
Jul 17 12:43:31.496642 ip-10-0-131-210 kubenswrapper[2569]: E0717 12:43:31.496612    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd5ae845bc7206bed804b1e78a846ead4e21758b76778dc655f47120fd95228c\": container with ID starting with cd5ae845bc7206bed804b1e78a846ead4e21758b76778dc655f47120fd95228c not found: ID does not exist" containerID="cd5ae845bc7206bed804b1e78a846ead4e21758b76778dc655f47120fd95228c"
Jul 17 12:43:31.496788 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:31.496650    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd5ae845bc7206bed804b1e78a846ead4e21758b76778dc655f47120fd95228c"} err="failed to get container status \"cd5ae845bc7206bed804b1e78a846ead4e21758b76778dc655f47120fd95228c\": rpc error: code = NotFound desc = could not find container \"cd5ae845bc7206bed804b1e78a846ead4e21758b76778dc655f47120fd95228c\": container with ID starting with cd5ae845bc7206bed804b1e78a846ead4e21758b76778dc655f47120fd95228c not found: ID does not exist"
Jul 17 12:43:31.515448 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:31.515420    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ddcf4dc7-zcr45"]
Jul 17 12:43:31.522339 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:31.522317    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6ddcf4dc7-zcr45"]
Jul 17 12:43:31.634900 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:43:31.634875    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cdf38e6-ee16-4216-8a1f-209c9a15a811" path="/var/lib/kubelet/pods/6cdf38e6-ee16-4216-8a1f-209c9a15a811/volumes"
Jul 17 12:44:31.696809 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:31.696730    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:44:31.697213 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:31.697076    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cdf38e6-ee16-4216-8a1f-209c9a15a811" containerName="console"
Jul 17 12:44:31.697213 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:31.697089    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cdf38e6-ee16-4216-8a1f-209c9a15a811" containerName="console"
Jul 17 12:44:31.697213 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:31.697154    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cdf38e6-ee16-4216-8a1f-209c9a15a811" containerName="console"
Jul 17 12:44:31.701271 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:31.701244    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-mx54h"
Jul 17 12:44:31.703887 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:31.703864    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 17 12:44:31.703987 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:31.703903    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 12:44:31.704883 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:31.704867    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mb4gp\""
Jul 17 12:44:31.704883 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:31.704879    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 12:44:31.710516 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:31.710494    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:44:31.739586 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:31.739565    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:44:31.800783 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:31.800740    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xfxm\" (UniqueName: \"kubernetes.io/projected/12782b8e-49c1-4930-b60e-d470c50b9bef-kube-api-access-8xfxm\") pod \"limitador-limitador-54d6d988d5-mx54h\" (UID: \"12782b8e-49c1-4930-b60e-d470c50b9bef\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mx54h"
Jul 17 12:44:31.800878 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:31.800798    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/12782b8e-49c1-4930-b60e-d470c50b9bef-config-file\") pod \"limitador-limitador-54d6d988d5-mx54h\" (UID: \"12782b8e-49c1-4930-b60e-d470c50b9bef\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mx54h"
Jul 17 12:44:31.901365 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:31.901337    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xfxm\" (UniqueName: \"kubernetes.io/projected/12782b8e-49c1-4930-b60e-d470c50b9bef-kube-api-access-8xfxm\") pod \"limitador-limitador-54d6d988d5-mx54h\" (UID: \"12782b8e-49c1-4930-b60e-d470c50b9bef\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mx54h"
Jul 17 12:44:31.901467 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:31.901376    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/12782b8e-49c1-4930-b60e-d470c50b9bef-config-file\") pod \"limitador-limitador-54d6d988d5-mx54h\" (UID: \"12782b8e-49c1-4930-b60e-d470c50b9bef\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mx54h"
Jul 17 12:44:31.901993 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:31.901973    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/12782b8e-49c1-4930-b60e-d470c50b9bef-config-file\") pod \"limitador-limitador-54d6d988d5-mx54h\" (UID: \"12782b8e-49c1-4930-b60e-d470c50b9bef\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mx54h"
Jul 17 12:44:31.910306 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:31.910288    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xfxm\" (UniqueName: \"kubernetes.io/projected/12782b8e-49c1-4930-b60e-d470c50b9bef-kube-api-access-8xfxm\") pod \"limitador-limitador-54d6d988d5-mx54h\" (UID: \"12782b8e-49c1-4930-b60e-d470c50b9bef\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-mx54h"
Jul 17 12:44:32.011931 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:32.011907    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-mx54h"
Jul 17 12:44:32.140481 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:32.140459    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:44:32.143464 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:44:32.143436    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12782b8e_49c1_4930_b60e_d470c50b9bef.slice/crio-b9460278165ecbcc27e0c501b2d21102b1dfe67844d81e11117919fd59a15b83 WatchSource:0}: Error finding container b9460278165ecbcc27e0c501b2d21102b1dfe67844d81e11117919fd59a15b83: Status 404 returned error can't find the container with id b9460278165ecbcc27e0c501b2d21102b1dfe67844d81e11117919fd59a15b83
Jul 17 12:44:32.763041 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:32.763010    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-mx54h" event={"ID":"12782b8e-49c1-4930-b60e-d470c50b9bef","Type":"ContainerStarted","Data":"b9460278165ecbcc27e0c501b2d21102b1dfe67844d81e11117919fd59a15b83"}
Jul 17 12:44:35.778158 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:35.778123    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-mx54h" event={"ID":"12782b8e-49c1-4930-b60e-d470c50b9bef","Type":"ContainerStarted","Data":"c7b6652ca6e719833b4d7971bce9f9ac9b95199abb5a54ef5676db15a9227161"}
Jul 17 12:44:35.778556 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:35.778211    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-mx54h"
Jul 17 12:44:35.794583 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:35.794539    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-mx54h" podStartSLOduration=2.135015144 podStartE2EDuration="4.794523162s" totalImagesPullingTime="2.659508018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:44:35.793467146 +0000 UTC m=+1350.797514031" watchObservedRunningTime="2026-07-17 12:44:35.794523162 +0000 UTC m=+1350.798570043"
Jul 17 12:44:46.783290 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:44:46.783261    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-mx54h"
Jul 17 12:45:49.240962 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:45:49.240935    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:46:11.901831 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:11.901742    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:46:50.654700 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:50.654663    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:46:54.761581 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:54.761547    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:46:58.801318 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.801285    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"]
Jul 17 12:46:58.804616 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.804594    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:58.808954 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.808926    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 17 12:46:58.809199 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.809176    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hdpj4\""
Jul 17 12:46:58.809288 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.809230    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 17 12:46:58.809288 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.809281    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 17 12:46:58.814165 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.814144    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"]
Jul 17 12:46:58.894065 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.894040    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/179bc674-34f1-4cec-94b5-e6f3b48fda3f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj\" (UID: \"179bc674-34f1-4cec-94b5-e6f3b48fda3f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:58.894182 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.894078    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/179bc674-34f1-4cec-94b5-e6f3b48fda3f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj\" (UID: \"179bc674-34f1-4cec-94b5-e6f3b48fda3f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:58.894182 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.894102    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/179bc674-34f1-4cec-94b5-e6f3b48fda3f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj\" (UID: \"179bc674-34f1-4cec-94b5-e6f3b48fda3f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:58.894182 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.894164    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/179bc674-34f1-4cec-94b5-e6f3b48fda3f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj\" (UID: \"179bc674-34f1-4cec-94b5-e6f3b48fda3f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:58.894296 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.894188    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/179bc674-34f1-4cec-94b5-e6f3b48fda3f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj\" (UID: \"179bc674-34f1-4cec-94b5-e6f3b48fda3f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:58.894296 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.894212    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnbd2\" (UniqueName: \"kubernetes.io/projected/179bc674-34f1-4cec-94b5-e6f3b48fda3f-kube-api-access-xnbd2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj\" (UID: \"179bc674-34f1-4cec-94b5-e6f3b48fda3f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:58.995418 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.995395    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/179bc674-34f1-4cec-94b5-e6f3b48fda3f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj\" (UID: \"179bc674-34f1-4cec-94b5-e6f3b48fda3f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:58.995513 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.995429    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/179bc674-34f1-4cec-94b5-e6f3b48fda3f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj\" (UID: \"179bc674-34f1-4cec-94b5-e6f3b48fda3f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:58.995513 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.995452    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/179bc674-34f1-4cec-94b5-e6f3b48fda3f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj\" (UID: \"179bc674-34f1-4cec-94b5-e6f3b48fda3f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:58.995513 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.995493    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/179bc674-34f1-4cec-94b5-e6f3b48fda3f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj\" (UID: \"179bc674-34f1-4cec-94b5-e6f3b48fda3f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:58.995626 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.995549    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/179bc674-34f1-4cec-94b5-e6f3b48fda3f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj\" (UID: \"179bc674-34f1-4cec-94b5-e6f3b48fda3f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:58.995626 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.995598    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnbd2\" (UniqueName: \"kubernetes.io/projected/179bc674-34f1-4cec-94b5-e6f3b48fda3f-kube-api-access-xnbd2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj\" (UID: \"179bc674-34f1-4cec-94b5-e6f3b48fda3f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:58.995899 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.995852    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/179bc674-34f1-4cec-94b5-e6f3b48fda3f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj\" (UID: \"179bc674-34f1-4cec-94b5-e6f3b48fda3f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:58.996024 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.995981    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/179bc674-34f1-4cec-94b5-e6f3b48fda3f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj\" (UID: \"179bc674-34f1-4cec-94b5-e6f3b48fda3f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:58.996086 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.996026    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/179bc674-34f1-4cec-94b5-e6f3b48fda3f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj\" (UID: \"179bc674-34f1-4cec-94b5-e6f3b48fda3f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:58.997857 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.997836    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/179bc674-34f1-4cec-94b5-e6f3b48fda3f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj\" (UID: \"179bc674-34f1-4cec-94b5-e6f3b48fda3f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:58.997946 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:58.997840    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/179bc674-34f1-4cec-94b5-e6f3b48fda3f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj\" (UID: \"179bc674-34f1-4cec-94b5-e6f3b48fda3f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:59.006488 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:59.006464    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnbd2\" (UniqueName: \"kubernetes.io/projected/179bc674-34f1-4cec-94b5-e6f3b48fda3f-kube-api-access-xnbd2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj\" (UID: \"179bc674-34f1-4cec-94b5-e6f3b48fda3f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:59.115368 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:59.115301    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:46:59.243161 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:59.243135    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"]
Jul 17 12:46:59.245686 ip-10-0-131-210 kubenswrapper[2569]: W0717 12:46:59.245657    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179bc674_34f1_4cec_94b5_e6f3b48fda3f.slice/crio-b20e92f318a229aa80c57ae355897b6fe09c6f2ad44a033599e50531d77f10c1 WatchSource:0}: Error finding container b20e92f318a229aa80c57ae355897b6fe09c6f2ad44a033599e50531d77f10c1: Status 404 returned error can't find the container with id b20e92f318a229aa80c57ae355897b6fe09c6f2ad44a033599e50531d77f10c1
Jul 17 12:46:59.247022 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:59.247003    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 12:46:59.433895 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:46:59.433808    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj" event={"ID":"179bc674-34f1-4cec-94b5-e6f3b48fda3f","Type":"ContainerStarted","Data":"b20e92f318a229aa80c57ae355897b6fe09c6f2ad44a033599e50531d77f10c1"}
Jul 17 12:47:00.457700 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:47:00.457667    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:47:01.445353 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:47:01.445310    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj" event={"ID":"179bc674-34f1-4cec-94b5-e6f3b48fda3f","Type":"ContainerStarted","Data":"2547bacb650a800f2311859843bc1ee59f015361c60c754deee2a44a90708898"}
Jul 17 12:47:01.463871 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:47:01.463814    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj" podStartSLOduration=2.120030236 podStartE2EDuration="3.463798898s" totalImagesPullingTime="1.343768662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:47:01.462934207 +0000 UTC m=+1496.466981087" watchObservedRunningTime="2026-07-17 12:47:01.463798898 +0000 UTC m=+1496.467845779"
Jul 17 12:47:05.765880 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:47:05.765833    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:47:09.115912 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:47:09.115873    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:47:09.116331 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:47:09.115930    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:47:09.128165 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:47:09.128136    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:47:09.491342 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:47:09.491314    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj"
Jul 17 12:47:13.278498 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:47:13.278387    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:47:13.295290 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:47:13.278685    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:47:13.662305 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:47:13.662233    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:49:02.468528 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:49:02.468438    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:49:12.112843 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:49:12.112799    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:49:24.005000 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:49:24.004960    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:51:06.466900 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:51:06.466860    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:51:47.266149 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:51:47.266114    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:51:56.600913 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:51:56.600873    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:52:05.087200 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:52:05.087167    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:52:13.308285 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:52:13.308167    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:52:13.316852 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:52:13.309527    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:52:15.667224 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:52:15.667191    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:52:25.671091 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:52:25.671055    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:52:37.238535 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:52:37.238500    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:52:47.972841 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:52:47.972808    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:52:55.170381 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:52:55.170340    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:53:02.704643 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:53:02.704599    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:53:11.271850 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:53:11.271812    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:53:23.763464 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:53:23.763431    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:53:33.669582 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:53:33.669540    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:53:58.107597 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:53:58.107558    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:54:08.473140 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:54:08.473099    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:54:24.665148 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:54:24.665112    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:54:35.962483 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:54:35.962440    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:54:45.259541 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:54:45.259494    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:54:54.266928 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:54:54.266890    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:55:02.765430 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:55:02.765340    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:56:50.466939 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:56:50.466900    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:56:59.069234 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:56:59.069203    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:57:13.343626 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:57:13.343513    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:57:13.347644 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:57:13.347536    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 12:57:13.765076 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:57:13.765048    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:57:23.861085 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:57:23.861057    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:57:34.262923 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:57:34.262885    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:57:42.665571 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:57:42.665530    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:57:52.568052 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:57:52.568006    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:58:01.755781 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:58:01.755727    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:58:12.668575 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:58:12.668542    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:58:22.062219 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:58:22.062185    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:58:33.760823 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:58:33.760788    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:58:40.964985 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:58:40.964948    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:58:59.875182 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:58:59.875130    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:59:08.765525 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:59:08.765490    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:59:19.465900 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:59:19.465856    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:59:29.867353 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:59:29.867267    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 12:59:47.867033 ip-10-0-131-210 kubenswrapper[2569]: I0717 12:59:47.866994    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:00:01.770101 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:00:01.770073    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:00:11.766372 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:00:11.766334    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:00:25.071829 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:00:25.071797    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:00:34.395717 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:00:34.395689    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:00:48.188812 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:00:48.188772    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:00:57.179870 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:00:57.179833    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:01:07.384149 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:01:07.384115    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:01:17.571072 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:01:17.571035    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:01:28.675223 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:01:28.675191    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:01:35.562688 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:01:35.562655    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:01:54.167427 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:01:54.167393    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:02:03.974787 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:03.974740    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:02:13.368267 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:13.368166    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 13:02:13.373263 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:13.373241    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 13:02:13.577240 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:13.577215    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:02:24.204010 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:24.203969    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:02:38.670911 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:38.670874    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:02:41.258920 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:41.258869    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7"]
Jul 17 13:02:41.262631 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:41.262604    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7"
Jul 17 13:02:41.265592 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:41.265568    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:02:41.265700 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:41.265644    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:02:41.269660 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:41.269629    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7"]
Jul 17 13:02:41.424464 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:41.424436    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0033d776-6163-4db5-8cdf-29097f5ff95f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7\" (UID: \"0033d776-6163-4db5-8cdf-29097f5ff95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7"
Jul 17 13:02:41.525125 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:41.525059    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0033d776-6163-4db5-8cdf-29097f5ff95f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7\" (UID: \"0033d776-6163-4db5-8cdf-29097f5ff95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7"
Jul 17 13:02:41.525622 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:41.525605    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0033d776-6163-4db5-8cdf-29097f5ff95f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7\" (UID: \"0033d776-6163-4db5-8cdf-29097f5ff95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7"
Jul 17 13:02:41.575192 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:41.575169    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7"
Jul 17 13:02:41.910076 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:41.909980    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7"]
Jul 17 13:02:41.912666 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:02:41.912633    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0033d776_6163_4db5_8cdf_29097f5ff95f.slice/crio-8b6f6585b46c1a6973c3e785e3e52abed65a3cd0eea4b7b29de1a20d20f14b72 WatchSource:0}: Error finding container 8b6f6585b46c1a6973c3e785e3e52abed65a3cd0eea4b7b29de1a20d20f14b72: Status 404 returned error can't find the container with id 8b6f6585b46c1a6973c3e785e3e52abed65a3cd0eea4b7b29de1a20d20f14b72
Jul 17 13:02:41.914975 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:41.914956    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 13:02:42.670220 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:42.670186    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7" event={"ID":"0033d776-6163-4db5-8cdf-29097f5ff95f","Type":"ContainerStarted","Data":"8b6f6585b46c1a6973c3e785e3e52abed65a3cd0eea4b7b29de1a20d20f14b72"}
Jul 17 13:02:46.689697 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:46.689618    2569 generic.go:356] "Generic (PLEG): container finished" podID="0033d776-6163-4db5-8cdf-29097f5ff95f" containerID="2ee85b7fcc41141e772bfb2807b44161b0205612066f3a724800039678336c6f" exitCode=7
Jul 17 13:02:46.690148 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:46.689694    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7" event={"ID":"0033d776-6163-4db5-8cdf-29097f5ff95f","Type":"ContainerDied","Data":"2ee85b7fcc41141e772bfb2807b44161b0205612066f3a724800039678336c6f"}
Jul 17 13:02:46.690148 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:46.689915    2569 scope.go:117] "RemoveContainer" containerID="2ee85b7fcc41141e772bfb2807b44161b0205612066f3a724800039678336c6f"
Jul 17 13:02:47.695900 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:47.695832    2569 generic.go:356] "Generic (PLEG): container finished" podID="0033d776-6163-4db5-8cdf-29097f5ff95f" containerID="18c69edcdc3f7d15118f855c699b7e39180db1465fd65da97e6ca90937b23719" exitCode=7
Jul 17 13:02:47.695900 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:47.695895    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7" event={"ID":"0033d776-6163-4db5-8cdf-29097f5ff95f","Type":"ContainerDied","Data":"18c69edcdc3f7d15118f855c699b7e39180db1465fd65da97e6ca90937b23719"}
Jul 17 13:02:47.696275 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:47.695936    2569 scope.go:117] "RemoveContainer" containerID="2ee85b7fcc41141e772bfb2807b44161b0205612066f3a724800039678336c6f"
Jul 17 13:02:47.696275 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:47.696072    2569 scope.go:117] "RemoveContainer" containerID="18c69edcdc3f7d15118f855c699b7e39180db1465fd65da97e6ca90937b23719"
Jul 17 13:02:47.696349 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:02:47.696281    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7_odh-ai-gateway-infra(0033d776-6163-4db5-8cdf-29097f5ff95f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7" podUID="0033d776-6163-4db5-8cdf-29097f5ff95f"
Jul 17 13:02:48.702376 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:02:48.702345    2569 scope.go:117] "RemoveContainer" containerID="18c69edcdc3f7d15118f855c699b7e39180db1465fd65da97e6ca90937b23719"
Jul 17 13:02:48.702738 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:02:48.702530    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7_odh-ai-gateway-infra(0033d776-6163-4db5-8cdf-29097f5ff95f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7" podUID="0033d776-6163-4db5-8cdf-29097f5ff95f"
Jul 17 13:03:03.628254 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:03.628227    2569 scope.go:117] "RemoveContainer" containerID="18c69edcdc3f7d15118f855c699b7e39180db1465fd65da97e6ca90937b23719"
Jul 17 13:03:04.775142 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:04.775114    2569 generic.go:356] "Generic (PLEG): container finished" podID="0033d776-6163-4db5-8cdf-29097f5ff95f" containerID="3f285a3bec78f57cdc4d4a79f110644a21e43d717fdecea8478c33eb5a28a206" exitCode=7
Jul 17 13:03:04.775532 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:04.775170    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7" event={"ID":"0033d776-6163-4db5-8cdf-29097f5ff95f","Type":"ContainerDied","Data":"3f285a3bec78f57cdc4d4a79f110644a21e43d717fdecea8478c33eb5a28a206"}
Jul 17 13:03:04.775532 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:04.775231    2569 scope.go:117] "RemoveContainer" containerID="18c69edcdc3f7d15118f855c699b7e39180db1465fd65da97e6ca90937b23719"
Jul 17 13:03:04.775532 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:04.775462    2569 scope.go:117] "RemoveContainer" containerID="3f285a3bec78f57cdc4d4a79f110644a21e43d717fdecea8478c33eb5a28a206"
Jul 17 13:03:04.775660 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:03:04.775622    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7_odh-ai-gateway-infra(0033d776-6163-4db5-8cdf-29097f5ff95f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7" podUID="0033d776-6163-4db5-8cdf-29097f5ff95f"
Jul 17 13:03:05.802117 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:05.802084    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7"]
Jul 17 13:03:05.926681 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:05.926657    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7"
Jul 17 13:03:06.005575 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:06.005551    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0033d776-6163-4db5-8cdf-29097f5ff95f-maas-api-service-ca\") pod \"0033d776-6163-4db5-8cdf-29097f5ff95f\" (UID: \"0033d776-6163-4db5-8cdf-29097f5ff95f\") "
Jul 17 13:03:06.005857 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:06.005835    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0033d776-6163-4db5-8cdf-29097f5ff95f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0033d776-6163-4db5-8cdf-29097f5ff95f" (UID: "0033d776-6163-4db5-8cdf-29097f5ff95f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:03:06.106643 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:06.106590    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0033d776-6163-4db5-8cdf-29097f5ff95f-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:03:06.785990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:06.785955    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7" event={"ID":"0033d776-6163-4db5-8cdf-29097f5ff95f","Type":"ContainerDied","Data":"8b6f6585b46c1a6973c3e785e3e52abed65a3cd0eea4b7b29de1a20d20f14b72"}
Jul 17 13:03:06.786144 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:06.786023    2569 scope.go:117] "RemoveContainer" containerID="3f285a3bec78f57cdc4d4a79f110644a21e43d717fdecea8478c33eb5a28a206"
Jul 17 13:03:06.786144 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:06.785964    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7"
Jul 17 13:03:06.807353 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:06.807328    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7"]
Jul 17 13:03:06.813903 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:06.813882    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-rh6m7"]
Jul 17 13:03:07.634497 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:07.634468    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0033d776-6163-4db5-8cdf-29097f5ff95f" path="/var/lib/kubelet/pods/0033d776-6163-4db5-8cdf-29097f5ff95f/volumes"
Jul 17 13:03:41.307700 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:41.307670    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f"]
Jul 17 13:03:41.308154 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:41.307983    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0033d776-6163-4db5-8cdf-29097f5ff95f" containerName="revoke-keys"
Jul 17 13:03:41.308154 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:41.307994    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0033d776-6163-4db5-8cdf-29097f5ff95f" containerName="revoke-keys"
Jul 17 13:03:41.308154 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:41.308003    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0033d776-6163-4db5-8cdf-29097f5ff95f" containerName="revoke-keys"
Jul 17 13:03:41.308154 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:41.308008    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0033d776-6163-4db5-8cdf-29097f5ff95f" containerName="revoke-keys"
Jul 17 13:03:41.308154 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:41.308065    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0033d776-6163-4db5-8cdf-29097f5ff95f" containerName="revoke-keys"
Jul 17 13:03:41.308154 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:41.308074    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0033d776-6163-4db5-8cdf-29097f5ff95f" containerName="revoke-keys"
Jul 17 13:03:41.308154 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:41.308081    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0033d776-6163-4db5-8cdf-29097f5ff95f" containerName="revoke-keys"
Jul 17 13:03:41.310976 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:41.310958    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f"
Jul 17 13:03:41.313669 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:41.313649    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:03:41.313669 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:41.313657    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:03:41.318476 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:41.318453    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f"]
Jul 17 13:03:41.356488 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:41.356457    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd5a20e-acf1-44ec-ad57-fbaba608185e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f\" (UID: \"5fd5a20e-acf1-44ec-ad57-fbaba608185e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f"
Jul 17 13:03:41.457438 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:41.457414    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd5a20e-acf1-44ec-ad57-fbaba608185e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f\" (UID: \"5fd5a20e-acf1-44ec-ad57-fbaba608185e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f"
Jul 17 13:03:41.457966 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:41.457948    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd5a20e-acf1-44ec-ad57-fbaba608185e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f\" (UID: \"5fd5a20e-acf1-44ec-ad57-fbaba608185e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f"
Jul 17 13:03:41.621523 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:41.621471    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f"
Jul 17 13:03:41.952824 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:41.952792    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f"]
Jul 17 13:03:41.954814 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:03:41.954789    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd5a20e_acf1_44ec_ad57_fbaba608185e.slice/crio-6337a35d02eb036685c15266d93745063f1e103b0cd258af23e416a0849e619f WatchSource:0}: Error finding container 6337a35d02eb036685c15266d93745063f1e103b0cd258af23e416a0849e619f: Status 404 returned error can't find the container with id 6337a35d02eb036685c15266d93745063f1e103b0cd258af23e416a0849e619f
Jul 17 13:03:42.945009 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:42.944931    2569 generic.go:356] "Generic (PLEG): container finished" podID="5fd5a20e-acf1-44ec-ad57-fbaba608185e" containerID="a9d30b0705e9025dbd5ac87d19f5ad2de27baf24db6d31061200f17bed00da7e" exitCode=7
Jul 17 13:03:42.945343 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:42.945020    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f" event={"ID":"5fd5a20e-acf1-44ec-ad57-fbaba608185e","Type":"ContainerDied","Data":"a9d30b0705e9025dbd5ac87d19f5ad2de27baf24db6d31061200f17bed00da7e"}
Jul 17 13:03:42.945343 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:42.945082    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f" event={"ID":"5fd5a20e-acf1-44ec-ad57-fbaba608185e","Type":"ContainerStarted","Data":"6337a35d02eb036685c15266d93745063f1e103b0cd258af23e416a0849e619f"}
Jul 17 13:03:42.945343 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:42.945111    2569 scope.go:117] "RemoveContainer" containerID="a9d30b0705e9025dbd5ac87d19f5ad2de27baf24db6d31061200f17bed00da7e"
Jul 17 13:03:43.951411 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:43.951325    2569 generic.go:356] "Generic (PLEG): container finished" podID="5fd5a20e-acf1-44ec-ad57-fbaba608185e" containerID="0156877a0800774a35a3a4ab5f8b398e7e3410beb1d3d34baf1258b67bb42613" exitCode=7
Jul 17 13:03:43.951829 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:43.951408    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f" event={"ID":"5fd5a20e-acf1-44ec-ad57-fbaba608185e","Type":"ContainerDied","Data":"0156877a0800774a35a3a4ab5f8b398e7e3410beb1d3d34baf1258b67bb42613"}
Jul 17 13:03:43.951829 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:43.951458    2569 scope.go:117] "RemoveContainer" containerID="a9d30b0705e9025dbd5ac87d19f5ad2de27baf24db6d31061200f17bed00da7e"
Jul 17 13:03:43.951829 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:43.951616    2569 scope.go:117] "RemoveContainer" containerID="0156877a0800774a35a3a4ab5f8b398e7e3410beb1d3d34baf1258b67bb42613"
Jul 17 13:03:43.951829 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:03:43.951821    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f_odh-ai-gateway-infra(5fd5a20e-acf1-44ec-ad57-fbaba608185e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f" podUID="5fd5a20e-acf1-44ec-ad57-fbaba608185e"
Jul 17 13:03:44.957059 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:44.957032    2569 scope.go:117] "RemoveContainer" containerID="0156877a0800774a35a3a4ab5f8b398e7e3410beb1d3d34baf1258b67bb42613"
Jul 17 13:03:44.957434 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:03:44.957214    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f_odh-ai-gateway-infra(5fd5a20e-acf1-44ec-ad57-fbaba608185e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f" podUID="5fd5a20e-acf1-44ec-ad57-fbaba608185e"
Jul 17 13:03:58.628677 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:58.628609    2569 scope.go:117] "RemoveContainer" containerID="0156877a0800774a35a3a4ab5f8b398e7e3410beb1d3d34baf1258b67bb42613"
Jul 17 13:03:59.014682 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:59.014648    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f" event={"ID":"5fd5a20e-acf1-44ec-ad57-fbaba608185e","Type":"ContainerStarted","Data":"1d02be0401562e3be9140e145becfe0c29e2a3b2c186092379e6df77865db004"}
Jul 17 13:03:59.032780 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:03:59.032703    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f" podStartSLOduration=18.032689949 podStartE2EDuration="18.032689949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:03:59.031877448 +0000 UTC m=+2514.035924330" watchObservedRunningTime="2026-07-17 13:03:59.032689949 +0000 UTC m=+2514.036736821"
Jul 17 13:04:00.019959 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:00.019928    2569 generic.go:356] "Generic (PLEG): container finished" podID="5fd5a20e-acf1-44ec-ad57-fbaba608185e" containerID="1d02be0401562e3be9140e145becfe0c29e2a3b2c186092379e6df77865db004" exitCode=7
Jul 17 13:04:00.020307 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:00.019961    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f" event={"ID":"5fd5a20e-acf1-44ec-ad57-fbaba608185e","Type":"ContainerDied","Data":"1d02be0401562e3be9140e145becfe0c29e2a3b2c186092379e6df77865db004"}
Jul 17 13:04:00.020307 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:00.020004    2569 scope.go:117] "RemoveContainer" containerID="0156877a0800774a35a3a4ab5f8b398e7e3410beb1d3d34baf1258b67bb42613"
Jul 17 13:04:00.020307 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:00.020220    2569 scope.go:117] "RemoveContainer" containerID="1d02be0401562e3be9140e145becfe0c29e2a3b2c186092379e6df77865db004"
Jul 17 13:04:00.020422 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:04:00.020370    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f_odh-ai-gateway-infra(5fd5a20e-acf1-44ec-ad57-fbaba608185e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f" podUID="5fd5a20e-acf1-44ec-ad57-fbaba608185e"
Jul 17 13:04:00.048020 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:00.047997    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f"]
Jul 17 13:04:01.149023 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:01.149002    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f"
Jul 17 13:04:01.304087 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:01.304028    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd5a20e-acf1-44ec-ad57-fbaba608185e-maas-api-service-ca\") pod \"5fd5a20e-acf1-44ec-ad57-fbaba608185e\" (UID: \"5fd5a20e-acf1-44ec-ad57-fbaba608185e\") "
Jul 17 13:04:01.304366 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:01.304343    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd5a20e-acf1-44ec-ad57-fbaba608185e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5fd5a20e-acf1-44ec-ad57-fbaba608185e" (UID: "5fd5a20e-acf1-44ec-ad57-fbaba608185e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:04:01.404958 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:01.404931    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fd5a20e-acf1-44ec-ad57-fbaba608185e-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:04:02.030785 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:02.030766    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f"
Jul 17 13:04:02.030892 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:02.030778    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f" event={"ID":"5fd5a20e-acf1-44ec-ad57-fbaba608185e","Type":"ContainerDied","Data":"6337a35d02eb036685c15266d93745063f1e103b0cd258af23e416a0849e619f"}
Jul 17 13:04:02.030892 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:02.030828    2569 scope.go:117] "RemoveContainer" containerID="1d02be0401562e3be9140e145becfe0c29e2a3b2c186092379e6df77865db004"
Jul 17 13:04:02.047892 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:02.047865    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f"]
Jul 17 13:04:02.049677 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:02.049657    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cx92f"]
Jul 17 13:04:03.633741 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:03.633708    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fd5a20e-acf1-44ec-ad57-fbaba608185e" path="/var/lib/kubelet/pods/5fd5a20e-acf1-44ec-ad57-fbaba608185e/volumes"
Jul 17 13:04:41.356784 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.356735    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm"]
Jul 17 13:04:41.357248 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.357081    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd5a20e-acf1-44ec-ad57-fbaba608185e" containerName="revoke-keys"
Jul 17 13:04:41.357248 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.357094    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd5a20e-acf1-44ec-ad57-fbaba608185e" containerName="revoke-keys"
Jul 17 13:04:41.357248 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.357102    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd5a20e-acf1-44ec-ad57-fbaba608185e" containerName="revoke-keys"
Jul 17 13:04:41.357248 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.357107    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd5a20e-acf1-44ec-ad57-fbaba608185e" containerName="revoke-keys"
Jul 17 13:04:41.357248 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.357120    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0033d776-6163-4db5-8cdf-29097f5ff95f" containerName="revoke-keys"
Jul 17 13:04:41.357248 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.357125    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0033d776-6163-4db5-8cdf-29097f5ff95f" containerName="revoke-keys"
Jul 17 13:04:41.357248 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.357132    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd5a20e-acf1-44ec-ad57-fbaba608185e" containerName="revoke-keys"
Jul 17 13:04:41.357248 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.357137    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd5a20e-acf1-44ec-ad57-fbaba608185e" containerName="revoke-keys"
Jul 17 13:04:41.357248 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.357201    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd5a20e-acf1-44ec-ad57-fbaba608185e" containerName="revoke-keys"
Jul 17 13:04:41.357248 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.357209    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd5a20e-acf1-44ec-ad57-fbaba608185e" containerName="revoke-keys"
Jul 17 13:04:41.360312 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.360295    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm"
Jul 17 13:04:41.363249 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.363225    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:04:41.363367 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.363322    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:04:41.369264 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.369237    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm"]
Jul 17 13:04:41.413193 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.413166    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ba3185-16d2-4404-bb6f-e46ead7681de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm\" (UID: \"28ba3185-16d2-4404-bb6f-e46ead7681de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm"
Jul 17 13:04:41.514414 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.514390    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ba3185-16d2-4404-bb6f-e46ead7681de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm\" (UID: \"28ba3185-16d2-4404-bb6f-e46ead7681de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm"
Jul 17 13:04:41.515032 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.515015    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ba3185-16d2-4404-bb6f-e46ead7681de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm\" (UID: \"28ba3185-16d2-4404-bb6f-e46ead7681de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm"
Jul 17 13:04:41.671994 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.671938    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm"
Jul 17 13:04:41.797990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:41.797961    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm"]
Jul 17 13:04:41.802218 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:04:41.802188    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ba3185_16d2_4404_bb6f_e46ead7681de.slice/crio-c1777b9c57b5a9cdfb10a925be4ffc00e2c6f40d9155bf1e91e18824e5a724f2 WatchSource:0}: Error finding container c1777b9c57b5a9cdfb10a925be4ffc00e2c6f40d9155bf1e91e18824e5a724f2: Status 404 returned error can't find the container with id c1777b9c57b5a9cdfb10a925be4ffc00e2c6f40d9155bf1e91e18824e5a724f2
Jul 17 13:04:42.210845 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:42.210812    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm" event={"ID":"28ba3185-16d2-4404-bb6f-e46ead7681de","Type":"ContainerStarted","Data":"f8d8f95416f601624f70d1161cbe2b68c0dcf1893549d9ca5edb3f30c8048e35"}
Jul 17 13:04:42.210982 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:42.210863    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm" event={"ID":"28ba3185-16d2-4404-bb6f-e46ead7681de","Type":"ContainerStarted","Data":"c1777b9c57b5a9cdfb10a925be4ffc00e2c6f40d9155bf1e91e18824e5a724f2"}
Jul 17 13:04:42.229910 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:42.229855    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm" podStartSLOduration=1.229839697 podStartE2EDuration="1.229839697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:04:42.228322278 +0000 UTC m=+2557.232369162" watchObservedRunningTime="2026-07-17 13:04:42.229839697 +0000 UTC m=+2557.233886578"
Jul 17 13:04:43.216403 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:43.216364    2569 generic.go:356] "Generic (PLEG): container finished" podID="28ba3185-16d2-4404-bb6f-e46ead7681de" containerID="f8d8f95416f601624f70d1161cbe2b68c0dcf1893549d9ca5edb3f30c8048e35" exitCode=7
Jul 17 13:04:43.216830 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:43.216452    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm" event={"ID":"28ba3185-16d2-4404-bb6f-e46ead7681de","Type":"ContainerDied","Data":"f8d8f95416f601624f70d1161cbe2b68c0dcf1893549d9ca5edb3f30c8048e35"}
Jul 17 13:04:43.216830 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:43.216592    2569 scope.go:117] "RemoveContainer" containerID="f8d8f95416f601624f70d1161cbe2b68c0dcf1893549d9ca5edb3f30c8048e35"
Jul 17 13:04:44.223011 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:44.222947    2569 generic.go:356] "Generic (PLEG): container finished" podID="28ba3185-16d2-4404-bb6f-e46ead7681de" containerID="2131f28cc981013477da174e9d89c00caf457d37a65af9b028e66e3a980c6228" exitCode=7
Jul 17 13:04:44.223356 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:44.223013    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm" event={"ID":"28ba3185-16d2-4404-bb6f-e46ead7681de","Type":"ContainerDied","Data":"2131f28cc981013477da174e9d89c00caf457d37a65af9b028e66e3a980c6228"}
Jul 17 13:04:44.223356 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:44.223055    2569 scope.go:117] "RemoveContainer" containerID="f8d8f95416f601624f70d1161cbe2b68c0dcf1893549d9ca5edb3f30c8048e35"
Jul 17 13:04:44.223356 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:44.223186    2569 scope.go:117] "RemoveContainer" containerID="2131f28cc981013477da174e9d89c00caf457d37a65af9b028e66e3a980c6228"
Jul 17 13:04:44.223484 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:04:44.223388    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm_odh-ai-gateway-infra(28ba3185-16d2-4404-bb6f-e46ead7681de)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm" podUID="28ba3185-16d2-4404-bb6f-e46ead7681de"
Jul 17 13:04:45.229919 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:45.229893    2569 scope.go:117] "RemoveContainer" containerID="2131f28cc981013477da174e9d89c00caf457d37a65af9b028e66e3a980c6228"
Jul 17 13:04:45.230275 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:04:45.230061    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm_odh-ai-gateway-infra(28ba3185-16d2-4404-bb6f-e46ead7681de)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm" podUID="28ba3185-16d2-4404-bb6f-e46ead7681de"
Jul 17 13:04:55.631137 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:55.631102    2569 scope.go:117] "RemoveContainer" containerID="2131f28cc981013477da174e9d89c00caf457d37a65af9b028e66e3a980c6228"
Jul 17 13:04:56.281236 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:56.281208    2569 generic.go:356] "Generic (PLEG): container finished" podID="28ba3185-16d2-4404-bb6f-e46ead7681de" containerID="e6dfb00d690840189fd9b96e592c27110a0e5841562ec55550f7aebfa72103cb" exitCode=7
Jul 17 13:04:56.281362 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:56.281273    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm" event={"ID":"28ba3185-16d2-4404-bb6f-e46ead7681de","Type":"ContainerDied","Data":"e6dfb00d690840189fd9b96e592c27110a0e5841562ec55550f7aebfa72103cb"}
Jul 17 13:04:56.281426 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:56.281381    2569 scope.go:117] "RemoveContainer" containerID="2131f28cc981013477da174e9d89c00caf457d37a65af9b028e66e3a980c6228"
Jul 17 13:04:56.281675 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:56.281660    2569 scope.go:117] "RemoveContainer" containerID="e6dfb00d690840189fd9b96e592c27110a0e5841562ec55550f7aebfa72103cb"
Jul 17 13:04:56.281888 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:04:56.281868    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm_odh-ai-gateway-infra(28ba3185-16d2-4404-bb6f-e46ead7681de)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm" podUID="28ba3185-16d2-4404-bb6f-e46ead7681de"
Jul 17 13:04:57.326333 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:57.326299    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm"]
Jul 17 13:04:57.462327 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:57.462305    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm"
Jul 17 13:04:57.540504 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:57.540482    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ba3185-16d2-4404-bb6f-e46ead7681de-maas-api-service-ca\") pod \"28ba3185-16d2-4404-bb6f-e46ead7681de\" (UID: \"28ba3185-16d2-4404-bb6f-e46ead7681de\") "
Jul 17 13:04:57.540788 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:57.540769    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ba3185-16d2-4404-bb6f-e46ead7681de-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "28ba3185-16d2-4404-bb6f-e46ead7681de" (UID: "28ba3185-16d2-4404-bb6f-e46ead7681de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:04:57.641255 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:57.641236    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ba3185-16d2-4404-bb6f-e46ead7681de-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:04:58.293412 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:58.293385    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm" event={"ID":"28ba3185-16d2-4404-bb6f-e46ead7681de","Type":"ContainerDied","Data":"c1777b9c57b5a9cdfb10a925be4ffc00e2c6f40d9155bf1e91e18824e5a724f2"}
Jul 17 13:04:58.293412 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:58.293403    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm"
Jul 17 13:04:58.293625 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:58.293457    2569 scope.go:117] "RemoveContainer" containerID="e6dfb00d690840189fd9b96e592c27110a0e5841562ec55550f7aebfa72103cb"
Jul 17 13:04:58.312183 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:58.312157    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm"]
Jul 17 13:04:58.318419 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:58.318392    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-27xdm"]
Jul 17 13:04:59.633867 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:04:59.633832    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28ba3185-16d2-4404-bb6f-e46ead7681de" path="/var/lib/kubelet/pods/28ba3185-16d2-4404-bb6f-e46ead7681de/volumes"
Jul 17 13:05:31.401324 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.401239    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc"]
Jul 17 13:05:31.401800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.401596    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ba3185-16d2-4404-bb6f-e46ead7681de" containerName="revoke-keys"
Jul 17 13:05:31.401800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.401615    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ba3185-16d2-4404-bb6f-e46ead7681de" containerName="revoke-keys"
Jul 17 13:05:31.401800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.401651    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ba3185-16d2-4404-bb6f-e46ead7681de" containerName="revoke-keys"
Jul 17 13:05:31.401800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.401661    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ba3185-16d2-4404-bb6f-e46ead7681de" containerName="revoke-keys"
Jul 17 13:05:31.401800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.401671    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ba3185-16d2-4404-bb6f-e46ead7681de" containerName="revoke-keys"
Jul 17 13:05:31.401800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.401680    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ba3185-16d2-4404-bb6f-e46ead7681de" containerName="revoke-keys"
Jul 17 13:05:31.401800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.401777    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="28ba3185-16d2-4404-bb6f-e46ead7681de" containerName="revoke-keys"
Jul 17 13:05:31.401800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.401788    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd5a20e-acf1-44ec-ad57-fbaba608185e" containerName="revoke-keys"
Jul 17 13:05:31.401800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.401798    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="28ba3185-16d2-4404-bb6f-e46ead7681de" containerName="revoke-keys"
Jul 17 13:05:31.404736 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.404719    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc"
Jul 17 13:05:31.407696 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.407675    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:05:31.409377 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.409359    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:05:31.413123 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.413098    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc"]
Jul 17 13:05:31.582483 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.582459    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e0eb58-c99d-4279-a48e-42e0a365d271-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc\" (UID: \"f8e0eb58-c99d-4279-a48e-42e0a365d271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc"
Jul 17 13:05:31.683011 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.682951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e0eb58-c99d-4279-a48e-42e0a365d271-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc\" (UID: \"f8e0eb58-c99d-4279-a48e-42e0a365d271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc"
Jul 17 13:05:31.683551 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.683531    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e0eb58-c99d-4279-a48e-42e0a365d271-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc\" (UID: \"f8e0eb58-c99d-4279-a48e-42e0a365d271\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc"
Jul 17 13:05:31.716121 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.716096    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc"
Jul 17 13:05:31.847344 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:31.847321    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc"]
Jul 17 13:05:31.849453 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:05:31.849425    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e0eb58_c99d_4279_a48e_42e0a365d271.slice/crio-c5ea102092ffcd6ba248e0770b30b632de4a3a275e1c1f9cd5525d7b56318e43 WatchSource:0}: Error finding container c5ea102092ffcd6ba248e0770b30b632de4a3a275e1c1f9cd5525d7b56318e43: Status 404 returned error can't find the container with id c5ea102092ffcd6ba248e0770b30b632de4a3a275e1c1f9cd5525d7b56318e43
Jul 17 13:05:32.464938 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:32.464912    2569 generic.go:356] "Generic (PLEG): container finished" podID="f8e0eb58-c99d-4279-a48e-42e0a365d271" containerID="6c8689ab5c490163d9445e345ce2a611c212d5706b2a6f95acb228a24c455496" exitCode=7
Jul 17 13:05:32.465249 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:32.464993    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc" event={"ID":"f8e0eb58-c99d-4279-a48e-42e0a365d271","Type":"ContainerDied","Data":"6c8689ab5c490163d9445e345ce2a611c212d5706b2a6f95acb228a24c455496"}
Jul 17 13:05:32.465249 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:32.465047    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc" event={"ID":"f8e0eb58-c99d-4279-a48e-42e0a365d271","Type":"ContainerStarted","Data":"c5ea102092ffcd6ba248e0770b30b632de4a3a275e1c1f9cd5525d7b56318e43"}
Jul 17 13:05:32.465249 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:32.465125    2569 scope.go:117] "RemoveContainer" containerID="6c8689ab5c490163d9445e345ce2a611c212d5706b2a6f95acb228a24c455496"
Jul 17 13:05:33.471943 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:33.471873    2569 generic.go:356] "Generic (PLEG): container finished" podID="f8e0eb58-c99d-4279-a48e-42e0a365d271" containerID="fa84c74f7cdda1067e1d0a03ddd75521ee2d13b95bd9e6dd63f2eb75a3217e54" exitCode=7
Jul 17 13:05:33.472341 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:33.471959    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc" event={"ID":"f8e0eb58-c99d-4279-a48e-42e0a365d271","Type":"ContainerDied","Data":"fa84c74f7cdda1067e1d0a03ddd75521ee2d13b95bd9e6dd63f2eb75a3217e54"}
Jul 17 13:05:33.472341 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:33.472051    2569 scope.go:117] "RemoveContainer" containerID="6c8689ab5c490163d9445e345ce2a611c212d5706b2a6f95acb228a24c455496"
Jul 17 13:05:33.472341 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:33.472131    2569 scope.go:117] "RemoveContainer" containerID="fa84c74f7cdda1067e1d0a03ddd75521ee2d13b95bd9e6dd63f2eb75a3217e54"
Jul 17 13:05:33.472478 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:05:33.472344    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc_odh-ai-gateway-infra(f8e0eb58-c99d-4279-a48e-42e0a365d271)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc" podUID="f8e0eb58-c99d-4279-a48e-42e0a365d271"
Jul 17 13:05:34.479148 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:34.479118    2569 scope.go:117] "RemoveContainer" containerID="fa84c74f7cdda1067e1d0a03ddd75521ee2d13b95bd9e6dd63f2eb75a3217e54"
Jul 17 13:05:34.479564 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:05:34.479293    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc_odh-ai-gateway-infra(f8e0eb58-c99d-4279-a48e-42e0a365d271)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc" podUID="f8e0eb58-c99d-4279-a48e-42e0a365d271"
Jul 17 13:05:44.278928 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:44.278895    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:05:48.628336 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:48.628306    2569 scope.go:117] "RemoveContainer" containerID="fa84c74f7cdda1067e1d0a03ddd75521ee2d13b95bd9e6dd63f2eb75a3217e54"
Jul 17 13:05:49.547707 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:49.547635    2569 generic.go:356] "Generic (PLEG): container finished" podID="f8e0eb58-c99d-4279-a48e-42e0a365d271" containerID="818c741c8025f30bc51636e615f42431b1a3f4f0b4cf421b5f629e5b7882799f" exitCode=6
Jul 17 13:05:49.547927 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:49.547710    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc" event={"ID":"f8e0eb58-c99d-4279-a48e-42e0a365d271","Type":"ContainerDied","Data":"818c741c8025f30bc51636e615f42431b1a3f4f0b4cf421b5f629e5b7882799f"}
Jul 17 13:05:49.547927 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:49.547783    2569 scope.go:117] "RemoveContainer" containerID="fa84c74f7cdda1067e1d0a03ddd75521ee2d13b95bd9e6dd63f2eb75a3217e54"
Jul 17 13:05:49.548090 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:49.548073    2569 scope.go:117] "RemoveContainer" containerID="818c741c8025f30bc51636e615f42431b1a3f4f0b4cf421b5f629e5b7882799f"
Jul 17 13:05:49.548311 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:05:49.548290    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc_odh-ai-gateway-infra(f8e0eb58-c99d-4279-a48e-42e0a365d271)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc" podUID="f8e0eb58-c99d-4279-a48e-42e0a365d271"
Jul 17 13:05:50.573428 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:50.573392    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc"]
Jul 17 13:05:50.712715 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:50.712694    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc"
Jul 17 13:05:50.724859 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:50.724837    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e0eb58-c99d-4279-a48e-42e0a365d271-maas-api-service-ca\") pod \"f8e0eb58-c99d-4279-a48e-42e0a365d271\" (UID: \"f8e0eb58-c99d-4279-a48e-42e0a365d271\") "
Jul 17 13:05:50.725221 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:50.725194    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e0eb58-c99d-4279-a48e-42e0a365d271-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8e0eb58-c99d-4279-a48e-42e0a365d271" (UID: "f8e0eb58-c99d-4279-a48e-42e0a365d271"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:05:50.825857 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:50.825801    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e0eb58-c99d-4279-a48e-42e0a365d271-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:05:51.561255 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:51.561226    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc"
Jul 17 13:05:51.561402 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:51.561254    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc" event={"ID":"f8e0eb58-c99d-4279-a48e-42e0a365d271","Type":"ContainerDied","Data":"c5ea102092ffcd6ba248e0770b30b632de4a3a275e1c1f9cd5525d7b56318e43"}
Jul 17 13:05:51.561402 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:51.561315    2569 scope.go:117] "RemoveContainer" containerID="818c741c8025f30bc51636e615f42431b1a3f4f0b4cf421b5f629e5b7882799f"
Jul 17 13:05:51.588037 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:51.588009    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc"]
Jul 17 13:05:51.595987 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:51.595965    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h7vrc"]
Jul 17 13:05:51.633781 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:51.633737    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8e0eb58-c99d-4279-a48e-42e0a365d271" path="/var/lib/kubelet/pods/f8e0eb58-c99d-4279-a48e-42e0a365d271/volumes"
Jul 17 13:05:55.893925 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:55.893893    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"]
Jul 17 13:05:55.894319 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:55.894203    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e0eb58-c99d-4279-a48e-42e0a365d271" containerName="revoke-keys"
Jul 17 13:05:55.894319 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:55.894213    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e0eb58-c99d-4279-a48e-42e0a365d271" containerName="revoke-keys"
Jul 17 13:05:55.894319 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:55.894233    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e0eb58-c99d-4279-a48e-42e0a365d271" containerName="revoke-keys"
Jul 17 13:05:55.894319 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:55.894238    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e0eb58-c99d-4279-a48e-42e0a365d271" containerName="revoke-keys"
Jul 17 13:05:55.894319 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:55.894287    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="28ba3185-16d2-4404-bb6f-e46ead7681de" containerName="revoke-keys"
Jul 17 13:05:55.894319 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:55.894296    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e0eb58-c99d-4279-a48e-42e0a365d271" containerName="revoke-keys"
Jul 17 13:05:55.894319 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:55.894302    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e0eb58-c99d-4279-a48e-42e0a365d271" containerName="revoke-keys"
Jul 17 13:05:55.898581 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:55.898558    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:05:55.901368 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:55.901347    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-359d73d8-serving-cert\""
Jul 17 13:05:55.901368 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:55.901356    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 13:05:55.902815 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:55.902792    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lwvzj\""
Jul 17 13:05:55.902926 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:55.902857    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:05:55.909396 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:55.909374    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"]
Jul 17 13:05:55.966787 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:55.966743    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bz2z\" (UniqueName: \"kubernetes.io/projected/c8e30576-e46b-4760-a491-9ddebcb0f49b-kube-api-access-2bz2z\") pod \"maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv\" (UID: \"c8e30576-e46b-4760-a491-9ddebcb0f49b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:05:55.966881 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:55.966807    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e30576-e46b-4760-a491-9ddebcb0f49b-openshift-service-ca\") pod \"maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv\" (UID: \"c8e30576-e46b-4760-a491-9ddebcb0f49b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:05:55.966881 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:55.966847    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8e30576-e46b-4760-a491-9ddebcb0f49b-maas-api-tls\") pod \"maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv\" (UID: \"c8e30576-e46b-4760-a491-9ddebcb0f49b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:05:56.067445 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:56.067417    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8e30576-e46b-4760-a491-9ddebcb0f49b-maas-api-tls\") pod \"maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv\" (UID: \"c8e30576-e46b-4760-a491-9ddebcb0f49b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:05:56.067538 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:56.067485    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bz2z\" (UniqueName: \"kubernetes.io/projected/c8e30576-e46b-4760-a491-9ddebcb0f49b-kube-api-access-2bz2z\") pod \"maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv\" (UID: \"c8e30576-e46b-4760-a491-9ddebcb0f49b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:05:56.067538 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:56.067506    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e30576-e46b-4760-a491-9ddebcb0f49b-openshift-service-ca\") pod \"maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv\" (UID: \"c8e30576-e46b-4760-a491-9ddebcb0f49b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:05:56.068110 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:56.068093    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e30576-e46b-4760-a491-9ddebcb0f49b-openshift-service-ca\") pod \"maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv\" (UID: \"c8e30576-e46b-4760-a491-9ddebcb0f49b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:05:56.069674 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:56.069657    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8e30576-e46b-4760-a491-9ddebcb0f49b-maas-api-tls\") pod \"maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv\" (UID: \"c8e30576-e46b-4760-a491-9ddebcb0f49b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:05:56.077058 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:56.077038    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bz2z\" (UniqueName: \"kubernetes.io/projected/c8e30576-e46b-4760-a491-9ddebcb0f49b-kube-api-access-2bz2z\") pod \"maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv\" (UID: \"c8e30576-e46b-4760-a491-9ddebcb0f49b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:05:56.209792 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:56.209736    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:05:56.346626 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:56.346599    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"]
Jul 17 13:05:56.348062 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:05:56.348035    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e30576_e46b_4760_a491_9ddebcb0f49b.slice/crio-6eeeaea41bc3b5c930859e1dd1a892c4dc487be26c35b4683d91102d63956f36 WatchSource:0}: Error finding container 6eeeaea41bc3b5c930859e1dd1a892c4dc487be26c35b4683d91102d63956f36: Status 404 returned error can't find the container with id 6eeeaea41bc3b5c930859e1dd1a892c4dc487be26c35b4683d91102d63956f36
Jul 17 13:05:56.585440 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:56.585409    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" event={"ID":"c8e30576-e46b-4760-a491-9ddebcb0f49b","Type":"ContainerStarted","Data":"6eeeaea41bc3b5c930859e1dd1a892c4dc487be26c35b4683d91102d63956f36"}
Jul 17 13:05:58.380596 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:58.380564    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:05:58.596862 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:58.596825    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" event={"ID":"c8e30576-e46b-4760-a491-9ddebcb0f49b","Type":"ContainerStarted","Data":"2b3cbc9661cbacfb76fec86ba2af751ddda864ad037833348a13a6a72e8a78af"}
Jul 17 13:05:58.596993 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:58.596915    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:05:58.618560 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:58.618513    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" podStartSLOduration=1.5676355960000001 podStartE2EDuration="3.618499804s" totalImagesPullingTime="2.050864208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:05:58.616235249 +0000 UTC m=+2633.620282132" watchObservedRunningTime="2026-07-17 13:05:58.618499804 +0000 UTC m=+2633.622546701"
Jul 17 13:05:59.602816 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:59.602783    2569 generic.go:356] "Generic (PLEG): container finished" podID="c8e30576-e46b-4760-a491-9ddebcb0f49b" containerID="2b3cbc9661cbacfb76fec86ba2af751ddda864ad037833348a13a6a72e8a78af" exitCode=1
Jul 17 13:05:59.603169 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:59.602859    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" event={"ID":"c8e30576-e46b-4760-a491-9ddebcb0f49b","Type":"ContainerDied","Data":"2b3cbc9661cbacfb76fec86ba2af751ddda864ad037833348a13a6a72e8a78af"}
Jul 17 13:05:59.603169 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:05:59.603050    2569 scope.go:117] "RemoveContainer" containerID="2b3cbc9661cbacfb76fec86ba2af751ddda864ad037833348a13a6a72e8a78af"
Jul 17 13:06:00.214046 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:00.214021    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf"]
Jul 17 13:06:00.214347 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:00.214335    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e0eb58-c99d-4279-a48e-42e0a365d271" containerName="revoke-keys"
Jul 17 13:06:00.214347 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:00.214348    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e0eb58-c99d-4279-a48e-42e0a365d271" containerName="revoke-keys"
Jul 17 13:06:00.214432 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:00.214420    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e0eb58-c99d-4279-a48e-42e0a365d271" containerName="revoke-keys"
Jul 17 13:06:00.218721 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:00.218706    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf"
Jul 17 13:06:00.221406 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:00.221384    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:06:00.227637 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:00.227608    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf"]
Jul 17 13:06:00.298567 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:00.298537    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180bb1a4-433a-4c64-bb7e-9b1e6321ddb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf\" (UID: \"180bb1a4-433a-4c64-bb7e-9b1e6321ddb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf"
Jul 17 13:06:00.398881 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:00.398853    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180bb1a4-433a-4c64-bb7e-9b1e6321ddb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf\" (UID: \"180bb1a4-433a-4c64-bb7e-9b1e6321ddb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf"
Jul 17 13:06:00.399473 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:00.399456    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180bb1a4-433a-4c64-bb7e-9b1e6321ddb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf\" (UID: \"180bb1a4-433a-4c64-bb7e-9b1e6321ddb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf"
Jul 17 13:06:00.530426 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:00.530408    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf"
Jul 17 13:06:00.609419 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:00.609387    2569 generic.go:356] "Generic (PLEG): container finished" podID="c8e30576-e46b-4760-a491-9ddebcb0f49b" containerID="4ac1d628683fbe87f79ee7d96128ed9eaa21de84c142e62676e21a4763442e37" exitCode=1
Jul 17 13:06:00.609812 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:00.609462    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" event={"ID":"c8e30576-e46b-4760-a491-9ddebcb0f49b","Type":"ContainerDied","Data":"4ac1d628683fbe87f79ee7d96128ed9eaa21de84c142e62676e21a4763442e37"}
Jul 17 13:06:00.609812 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:00.609517    2569 scope.go:117] "RemoveContainer" containerID="2b3cbc9661cbacfb76fec86ba2af751ddda864ad037833348a13a6a72e8a78af"
Jul 17 13:06:00.609812 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:00.609746    2569 scope.go:117] "RemoveContainer" containerID="4ac1d628683fbe87f79ee7d96128ed9eaa21de84c142e62676e21a4763442e37"
Jul 17 13:06:00.610023 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:06:00.609999    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv_odh-ai-gateway-infra(c8e30576-e46b-4760-a491-9ddebcb0f49b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b"
Jul 17 13:06:00.673313 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:00.673290    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf"]
Jul 17 13:06:00.675792 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:06:00.675740    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180bb1a4_433a_4c64_bb7e_9b1e6321ddb3.slice/crio-1b89d2990ceaa3437241093c2c3cb35e47025bd1a8c9497c1e94e738182512fb WatchSource:0}: Error finding container 1b89d2990ceaa3437241093c2c3cb35e47025bd1a8c9497c1e94e738182512fb: Status 404 returned error can't find the container with id 1b89d2990ceaa3437241093c2c3cb35e47025bd1a8c9497c1e94e738182512fb
Jul 17 13:06:01.616170 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:01.616135    2569 generic.go:356] "Generic (PLEG): container finished" podID="180bb1a4-433a-4c64-bb7e-9b1e6321ddb3" containerID="003e89f791a49356ea3c54432c3039a6cac521eba5285115ee1aab047d6c4e9e" exitCode=7
Jul 17 13:06:01.616548 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:01.616227    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf" event={"ID":"180bb1a4-433a-4c64-bb7e-9b1e6321ddb3","Type":"ContainerDied","Data":"003e89f791a49356ea3c54432c3039a6cac521eba5285115ee1aab047d6c4e9e"}
Jul 17 13:06:01.616548 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:01.616286    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf" event={"ID":"180bb1a4-433a-4c64-bb7e-9b1e6321ddb3","Type":"ContainerStarted","Data":"1b89d2990ceaa3437241093c2c3cb35e47025bd1a8c9497c1e94e738182512fb"}
Jul 17 13:06:01.616548 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:01.616383    2569 scope.go:117] "RemoveContainer" containerID="003e89f791a49356ea3c54432c3039a6cac521eba5285115ee1aab047d6c4e9e"
Jul 17 13:06:01.618337 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:01.618319    2569 scope.go:117] "RemoveContainer" containerID="4ac1d628683fbe87f79ee7d96128ed9eaa21de84c142e62676e21a4763442e37"
Jul 17 13:06:01.618503 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:06:01.618477    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv_odh-ai-gateway-infra(c8e30576-e46b-4760-a491-9ddebcb0f49b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b"
Jul 17 13:06:02.624331 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:02.624296    2569 generic.go:356] "Generic (PLEG): container finished" podID="180bb1a4-433a-4c64-bb7e-9b1e6321ddb3" containerID="5c94cbe1a2df8507a36b49f202f126f72f8426f15041444c77455e403e9b7630" exitCode=7
Jul 17 13:06:02.624659 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:02.624379    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf" event={"ID":"180bb1a4-433a-4c64-bb7e-9b1e6321ddb3","Type":"ContainerDied","Data":"5c94cbe1a2df8507a36b49f202f126f72f8426f15041444c77455e403e9b7630"}
Jul 17 13:06:02.624659 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:02.624438    2569 scope.go:117] "RemoveContainer" containerID="003e89f791a49356ea3c54432c3039a6cac521eba5285115ee1aab047d6c4e9e"
Jul 17 13:06:02.624659 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:02.624597    2569 scope.go:117] "RemoveContainer" containerID="5c94cbe1a2df8507a36b49f202f126f72f8426f15041444c77455e403e9b7630"
Jul 17 13:06:02.624819 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:06:02.624778    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf_odh-ai-gateway-infra(180bb1a4-433a-4c64-bb7e-9b1e6321ddb3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf" podUID="180bb1a4-433a-4c64-bb7e-9b1e6321ddb3"
Jul 17 13:06:03.597222 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:03.597189    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:06:03.597519 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:03.597506    2569 scope.go:117] "RemoveContainer" containerID="4ac1d628683fbe87f79ee7d96128ed9eaa21de84c142e62676e21a4763442e37"
Jul 17 13:06:03.597675 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:06:03.597658    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv_odh-ai-gateway-infra(c8e30576-e46b-4760-a491-9ddebcb0f49b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b"
Jul 17 13:06:03.635919 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:03.635863    2569 scope.go:117] "RemoveContainer" containerID="5c94cbe1a2df8507a36b49f202f126f72f8426f15041444c77455e403e9b7630"
Jul 17 13:06:03.636438 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:06:03.636405    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf_odh-ai-gateway-infra(180bb1a4-433a-4c64-bb7e-9b1e6321ddb3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf" podUID="180bb1a4-433a-4c64-bb7e-9b1e6321ddb3"
Jul 17 13:06:06.210150 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:06.210119    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:06:06.210498 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:06.210435    2569 scope.go:117] "RemoveContainer" containerID="4ac1d628683fbe87f79ee7d96128ed9eaa21de84c142e62676e21a4763442e37"
Jul 17 13:06:06.210611 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:06:06.210589    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv_odh-ai-gateway-infra(c8e30576-e46b-4760-a491-9ddebcb0f49b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b"
Jul 17 13:06:16.629180 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:16.629145    2569 scope.go:117] "RemoveContainer" containerID="5c94cbe1a2df8507a36b49f202f126f72f8426f15041444c77455e403e9b7630"
Jul 17 13:06:17.705200 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:17.705169    2569 generic.go:356] "Generic (PLEG): container finished" podID="180bb1a4-433a-4c64-bb7e-9b1e6321ddb3" containerID="9a9288b836b01617ee156d843359d47f6b8a0aee3b0fa252c234d428a1567e26" exitCode=7
Jul 17 13:06:17.705666 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:17.705212    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf" event={"ID":"180bb1a4-433a-4c64-bb7e-9b1e6321ddb3","Type":"ContainerDied","Data":"9a9288b836b01617ee156d843359d47f6b8a0aee3b0fa252c234d428a1567e26"}
Jul 17 13:06:17.705666 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:17.705297    2569 scope.go:117] "RemoveContainer" containerID="5c94cbe1a2df8507a36b49f202f126f72f8426f15041444c77455e403e9b7630"
Jul 17 13:06:17.705666 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:17.705526    2569 scope.go:117] "RemoveContainer" containerID="9a9288b836b01617ee156d843359d47f6b8a0aee3b0fa252c234d428a1567e26"
Jul 17 13:06:17.705810 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:06:17.705704    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf_odh-ai-gateway-infra(180bb1a4-433a-4c64-bb7e-9b1e6321ddb3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf" podUID="180bb1a4-433a-4c64-bb7e-9b1e6321ddb3"
Jul 17 13:06:18.737077 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:18.737044    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf"]
Jul 17 13:06:18.874690 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:18.874666    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf"
Jul 17 13:06:18.935742 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:18.935713    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180bb1a4-433a-4c64-bb7e-9b1e6321ddb3-maas-api-service-ca\") pod \"180bb1a4-433a-4c64-bb7e-9b1e6321ddb3\" (UID: \"180bb1a4-433a-4c64-bb7e-9b1e6321ddb3\") "
Jul 17 13:06:18.936027 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:18.936007    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/180bb1a4-433a-4c64-bb7e-9b1e6321ddb3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "180bb1a4-433a-4c64-bb7e-9b1e6321ddb3" (UID: "180bb1a4-433a-4c64-bb7e-9b1e6321ddb3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:06:19.036621 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:19.036567    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/180bb1a4-433a-4c64-bb7e-9b1e6321ddb3-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:06:19.717688 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:19.717658    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf" event={"ID":"180bb1a4-433a-4c64-bb7e-9b1e6321ddb3","Type":"ContainerDied","Data":"1b89d2990ceaa3437241093c2c3cb35e47025bd1a8c9497c1e94e738182512fb"}
Jul 17 13:06:19.717688 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:19.717686    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf"
Jul 17 13:06:19.717913 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:19.717722    2569 scope.go:117] "RemoveContainer" containerID="9a9288b836b01617ee156d843359d47f6b8a0aee3b0fa252c234d428a1567e26"
Jul 17 13:06:19.737102 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:19.737072    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf"]
Jul 17 13:06:19.745729 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:19.745705    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sfhpf"]
Jul 17 13:06:20.628347 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:20.628326    2569 scope.go:117] "RemoveContainer" containerID="4ac1d628683fbe87f79ee7d96128ed9eaa21de84c142e62676e21a4763442e37"
Jul 17 13:06:21.634069 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:21.634016    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="180bb1a4-433a-4c64-bb7e-9b1e6321ddb3" path="/var/lib/kubelet/pods/180bb1a4-433a-4c64-bb7e-9b1e6321ddb3/volumes"
Jul 17 13:06:21.731389 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:21.731363    2569 generic.go:356] "Generic (PLEG): container finished" podID="c8e30576-e46b-4760-a491-9ddebcb0f49b" containerID="21369b9222bfede2f3ba00489f2b320d9b570e08395c391526790c0e14430199" exitCode=1
Jul 17 13:06:21.731491 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:21.731434    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" event={"ID":"c8e30576-e46b-4760-a491-9ddebcb0f49b","Type":"ContainerDied","Data":"21369b9222bfede2f3ba00489f2b320d9b570e08395c391526790c0e14430199"}
Jul 17 13:06:21.731538 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:21.731498    2569 scope.go:117] "RemoveContainer" containerID="4ac1d628683fbe87f79ee7d96128ed9eaa21de84c142e62676e21a4763442e37"
Jul 17 13:06:21.731809 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:21.731792    2569 scope.go:117] "RemoveContainer" containerID="21369b9222bfede2f3ba00489f2b320d9b570e08395c391526790c0e14430199"
Jul 17 13:06:21.731989 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:06:21.731970    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv_odh-ai-gateway-infra(c8e30576-e46b-4760-a491-9ddebcb0f49b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b"
Jul 17 13:06:23.597910 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:23.597875    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:06:23.598290 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:23.598234    2569 scope.go:117] "RemoveContainer" containerID="21369b9222bfede2f3ba00489f2b320d9b570e08395c391526790c0e14430199"
Jul 17 13:06:23.598405 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:06:23.598383    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv_odh-ai-gateway-infra(c8e30576-e46b-4760-a491-9ddebcb0f49b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b"
Jul 17 13:06:26.210640 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:26.210602    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:06:26.211093 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:26.211030    2569 scope.go:117] "RemoveContainer" containerID="21369b9222bfede2f3ba00489f2b320d9b570e08395c391526790c0e14430199"
Jul 17 13:06:26.211233 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:06:26.211208    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv_odh-ai-gateway-infra(c8e30576-e46b-4760-a491-9ddebcb0f49b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b"
Jul 17 13:06:36.628825 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:36.628793    2569 scope.go:117] "RemoveContainer" containerID="21369b9222bfede2f3ba00489f2b320d9b570e08395c391526790c0e14430199"
Jul 17 13:06:36.629254 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:06:36.628968    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv_odh-ai-gateway-infra(c8e30576-e46b-4760-a491-9ddebcb0f49b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b"
Jul 17 13:06:48.628426 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:48.628393    2569 scope.go:117] "RemoveContainer" containerID="21369b9222bfede2f3ba00489f2b320d9b570e08395c391526790c0e14430199"
Jul 17 13:06:49.863009 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:49.862976    2569 generic.go:356] "Generic (PLEG): container finished" podID="c8e30576-e46b-4760-a491-9ddebcb0f49b" containerID="81eb371978a3d42a513cede9b46897e7a4ce6821fcbaf224747c478d7df00c8b" exitCode=1
Jul 17 13:06:49.863350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:49.863051    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" event={"ID":"c8e30576-e46b-4760-a491-9ddebcb0f49b","Type":"ContainerDied","Data":"81eb371978a3d42a513cede9b46897e7a4ce6821fcbaf224747c478d7df00c8b"}
Jul 17 13:06:49.863350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:49.863118    2569 scope.go:117] "RemoveContainer" containerID="21369b9222bfede2f3ba00489f2b320d9b570e08395c391526790c0e14430199"
Jul 17 13:06:49.863480 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:49.863454    2569 scope.go:117] "RemoveContainer" containerID="81eb371978a3d42a513cede9b46897e7a4ce6821fcbaf224747c478d7df00c8b"
Jul 17 13:06:49.863673 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:06:49.863651    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv_odh-ai-gateway-infra(c8e30576-e46b-4760-a491-9ddebcb0f49b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b"
Jul 17 13:06:50.272000 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:50.271969    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg"]
Jul 17 13:06:50.272338 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:50.272320    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="180bb1a4-433a-4c64-bb7e-9b1e6321ddb3" containerName="revoke-keys"
Jul 17 13:06:50.272418 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:50.272340    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="180bb1a4-433a-4c64-bb7e-9b1e6321ddb3" containerName="revoke-keys"
Jul 17 13:06:50.272418 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:50.272358    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="180bb1a4-433a-4c64-bb7e-9b1e6321ddb3" containerName="revoke-keys"
Jul 17 13:06:50.272418 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:50.272367    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="180bb1a4-433a-4c64-bb7e-9b1e6321ddb3" containerName="revoke-keys"
Jul 17 13:06:50.272557 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:50.272473    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="180bb1a4-433a-4c64-bb7e-9b1e6321ddb3" containerName="revoke-keys"
Jul 17 13:06:50.272557 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:50.272493    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="180bb1a4-433a-4c64-bb7e-9b1e6321ddb3" containerName="revoke-keys"
Jul 17 13:06:50.276654 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:50.276634    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg"
Jul 17 13:06:50.280015 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:50.279994    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:06:50.286828 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:50.286805    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg"]
Jul 17 13:06:50.370397 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:50.370373    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c211d417-f590-41fd-b593-28dcf4617240-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg\" (UID: \"c211d417-f590-41fd-b593-28dcf4617240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg"
Jul 17 13:06:50.471712 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:50.471685    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c211d417-f590-41fd-b593-28dcf4617240-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg\" (UID: \"c211d417-f590-41fd-b593-28dcf4617240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg"
Jul 17 13:06:50.472246 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:50.472228    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c211d417-f590-41fd-b593-28dcf4617240-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg\" (UID: \"c211d417-f590-41fd-b593-28dcf4617240\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg"
Jul 17 13:06:50.587341 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:50.587290    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg"
Jul 17 13:06:50.713638 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:50.713583    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg"]
Jul 17 13:06:50.715866 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:06:50.715835    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc211d417_f590_41fd_b593_28dcf4617240.slice/crio-fa2dcc6ced07d2cf057b5845bf316b644a844ee1947cc426f3600ebf34cdbab9 WatchSource:0}: Error finding container fa2dcc6ced07d2cf057b5845bf316b644a844ee1947cc426f3600ebf34cdbab9: Status 404 returned error can't find the container with id fa2dcc6ced07d2cf057b5845bf316b644a844ee1947cc426f3600ebf34cdbab9
Jul 17 13:06:50.868679 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:50.868627    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg" event={"ID":"c211d417-f590-41fd-b593-28dcf4617240","Type":"ContainerStarted","Data":"fa2dcc6ced07d2cf057b5845bf316b644a844ee1947cc426f3600ebf34cdbab9"}
Jul 17 13:06:51.875295 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:51.875261    2569 generic.go:356] "Generic (PLEG): container finished" podID="c211d417-f590-41fd-b593-28dcf4617240" containerID="464b9bf3aadf76605c1e956aa239494b06bb3781f721cc6edb4d86a8bf5439ba" exitCode=7
Jul 17 13:06:51.875791 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:51.875347    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg" event={"ID":"c211d417-f590-41fd-b593-28dcf4617240","Type":"ContainerDied","Data":"464b9bf3aadf76605c1e956aa239494b06bb3781f721cc6edb4d86a8bf5439ba"}
Jul 17 13:06:51.875791 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:51.875521    2569 scope.go:117] "RemoveContainer" containerID="464b9bf3aadf76605c1e956aa239494b06bb3781f721cc6edb4d86a8bf5439ba"
Jul 17 13:06:52.881791 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:52.881737    2569 generic.go:356] "Generic (PLEG): container finished" podID="c211d417-f590-41fd-b593-28dcf4617240" containerID="0ef5056b7c470a6e4ab486eaa8d0d0ddf0b74b89905121497f3d16a13971da91" exitCode=7
Jul 17 13:06:52.882143 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:52.881827    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg" event={"ID":"c211d417-f590-41fd-b593-28dcf4617240","Type":"ContainerDied","Data":"0ef5056b7c470a6e4ab486eaa8d0d0ddf0b74b89905121497f3d16a13971da91"}
Jul 17 13:06:52.882143 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:52.881892    2569 scope.go:117] "RemoveContainer" containerID="464b9bf3aadf76605c1e956aa239494b06bb3781f721cc6edb4d86a8bf5439ba"
Jul 17 13:06:52.882143 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:52.882085    2569 scope.go:117] "RemoveContainer" containerID="0ef5056b7c470a6e4ab486eaa8d0d0ddf0b74b89905121497f3d16a13971da91"
Jul 17 13:06:52.882396 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:06:52.882299    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg_odh-ai-gateway-infra(c211d417-f590-41fd-b593-28dcf4617240)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg" podUID="c211d417-f590-41fd-b593-28dcf4617240"
Jul 17 13:06:53.597551 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:53.597523    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:06:53.597900 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:53.597885    2569 scope.go:117] "RemoveContainer" containerID="81eb371978a3d42a513cede9b46897e7a4ce6821fcbaf224747c478d7df00c8b"
Jul 17 13:06:53.598050 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:06:53.598031    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv_odh-ai-gateway-infra(c8e30576-e46b-4760-a491-9ddebcb0f49b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b"
Jul 17 13:06:53.888958 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:53.888882    2569 scope.go:117] "RemoveContainer" containerID="0ef5056b7c470a6e4ab486eaa8d0d0ddf0b74b89905121497f3d16a13971da91"
Jul 17 13:06:53.889272 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:06:53.889051    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg_odh-ai-gateway-infra(c211d417-f590-41fd-b593-28dcf4617240)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg" podUID="c211d417-f590-41fd-b593-28dcf4617240"
Jul 17 13:06:56.210148 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:56.210114    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:06:56.210546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:06:56.210442    2569 scope.go:117] "RemoveContainer" containerID="81eb371978a3d42a513cede9b46897e7a4ce6821fcbaf224747c478d7df00c8b"
Jul 17 13:06:56.210620 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:06:56.210599    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv_odh-ai-gateway-infra(c8e30576-e46b-4760-a491-9ddebcb0f49b)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b"
Jul 17 13:07:03.172855 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:03.172739    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:07:04.628284 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:04.628259    2569 scope.go:117] "RemoveContainer" containerID="0ef5056b7c470a6e4ab486eaa8d0d0ddf0b74b89905121497f3d16a13971da91"
Jul 17 13:07:04.939913 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:04.939856    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg" event={"ID":"c211d417-f590-41fd-b593-28dcf4617240","Type":"ContainerStarted","Data":"fd70cdf408a1f20efa5010c8dded88a548b2f466f85653114cab719ae7a95fb8"}
Jul 17 13:07:04.967214 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:04.967170    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg" podStartSLOduration=14.967156184 podStartE2EDuration="14.967156184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:07:04.965995426 +0000 UTC m=+2699.970042308" watchObservedRunningTime="2026-07-17 13:07:04.967156184 +0000 UTC m=+2699.971203065"
Jul 17 13:07:05.497844 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.497771    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"]
Jul 17 13:07:05.626151 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.626123    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:07:05.789055 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.788990    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e30576-e46b-4760-a491-9ddebcb0f49b-openshift-service-ca\") pod \"c8e30576-e46b-4760-a491-9ddebcb0f49b\" (UID: \"c8e30576-e46b-4760-a491-9ddebcb0f49b\") "
Jul 17 13:07:05.789055 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.789037    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bz2z\" (UniqueName: \"kubernetes.io/projected/c8e30576-e46b-4760-a491-9ddebcb0f49b-kube-api-access-2bz2z\") pod \"c8e30576-e46b-4760-a491-9ddebcb0f49b\" (UID: \"c8e30576-e46b-4760-a491-9ddebcb0f49b\") "
Jul 17 13:07:05.789425 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.789068    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8e30576-e46b-4760-a491-9ddebcb0f49b-maas-api-tls\") pod \"c8e30576-e46b-4760-a491-9ddebcb0f49b\" (UID: \"c8e30576-e46b-4760-a491-9ddebcb0f49b\") "
Jul 17 13:07:05.789425 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.789316    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e30576-e46b-4760-a491-9ddebcb0f49b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8e30576-e46b-4760-a491-9ddebcb0f49b" (UID: "c8e30576-e46b-4760-a491-9ddebcb0f49b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:07:05.789425 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.789389    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e30576-e46b-4760-a491-9ddebcb0f49b-openshift-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:07:05.791174 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.791149    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e30576-e46b-4760-a491-9ddebcb0f49b-kube-api-access-2bz2z" (OuterVolumeSpecName: "kube-api-access-2bz2z") pod "c8e30576-e46b-4760-a491-9ddebcb0f49b" (UID: "c8e30576-e46b-4760-a491-9ddebcb0f49b"). InnerVolumeSpecName "kube-api-access-2bz2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:07:05.791274 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.791172    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8e30576-e46b-4760-a491-9ddebcb0f49b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8e30576-e46b-4760-a491-9ddebcb0f49b" (UID: "c8e30576-e46b-4760-a491-9ddebcb0f49b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:07:05.890101 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.890078    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bz2z\" (UniqueName: \"kubernetes.io/projected/c8e30576-e46b-4760-a491-9ddebcb0f49b-kube-api-access-2bz2z\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:07:05.890101 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.890100    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8e30576-e46b-4760-a491-9ddebcb0f49b-maas-api-tls\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:07:05.945186 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.945156    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv" event={"ID":"c8e30576-e46b-4760-a491-9ddebcb0f49b","Type":"ContainerDied","Data":"6eeeaea41bc3b5c930859e1dd1a892c4dc487be26c35b4683d91102d63956f36"}
Jul 17 13:07:05.945301 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.945184    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"
Jul 17 13:07:05.945301 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.945221    2569 scope.go:117] "RemoveContainer" containerID="81eb371978a3d42a513cede9b46897e7a4ce6821fcbaf224747c478d7df00c8b"
Jul 17 13:07:05.947179 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.947157    2569 generic.go:356] "Generic (PLEG): container finished" podID="c211d417-f590-41fd-b593-28dcf4617240" containerID="fd70cdf408a1f20efa5010c8dded88a548b2f466f85653114cab719ae7a95fb8" exitCode=7
Jul 17 13:07:05.947283 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.947188    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg" event={"ID":"c211d417-f590-41fd-b593-28dcf4617240","Type":"ContainerDied","Data":"fd70cdf408a1f20efa5010c8dded88a548b2f466f85653114cab719ae7a95fb8"}
Jul 17 13:07:05.947475 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.947462    2569 scope.go:117] "RemoveContainer" containerID="fd70cdf408a1f20efa5010c8dded88a548b2f466f85653114cab719ae7a95fb8"
Jul 17 13:07:05.947658 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:05.947641    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg_odh-ai-gateway-infra(c211d417-f590-41fd-b593-28dcf4617240)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg" podUID="c211d417-f590-41fd-b593-28dcf4617240"
Jul 17 13:07:05.955175 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.955155    2569 scope.go:117] "RemoveContainer" containerID="0ef5056b7c470a6e4ab486eaa8d0d0ddf0b74b89905121497f3d16a13971da91"
Jul 17 13:07:05.976383 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.976361    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg"]
Jul 17 13:07:05.981328 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.981307    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"]
Jul 17 13:07:05.987875 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:05.987856    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-359d73d8-55f987c8bd-rpmsv"]
Jul 17 13:07:07.079977 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:07.079956    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg"
Jul 17 13:07:07.201145 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:07.201120    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c211d417-f590-41fd-b593-28dcf4617240-maas-api-service-ca\") pod \"c211d417-f590-41fd-b593-28dcf4617240\" (UID: \"c211d417-f590-41fd-b593-28dcf4617240\") "
Jul 17 13:07:07.201447 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:07.201427    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c211d417-f590-41fd-b593-28dcf4617240-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c211d417-f590-41fd-b593-28dcf4617240" (UID: "c211d417-f590-41fd-b593-28dcf4617240"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:07:07.302474 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:07.302451    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c211d417-f590-41fd-b593-28dcf4617240-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:07:07.634511 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:07.634473    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b" path="/var/lib/kubelet/pods/c8e30576-e46b-4760-a491-9ddebcb0f49b/volumes"
Jul 17 13:07:07.961164 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:07.961104    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg"
Jul 17 13:07:07.961164 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:07.961122    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg" event={"ID":"c211d417-f590-41fd-b593-28dcf4617240","Type":"ContainerDied","Data":"fa2dcc6ced07d2cf057b5845bf316b644a844ee1947cc426f3600ebf34cdbab9"}
Jul 17 13:07:07.961349 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:07.961179    2569 scope.go:117] "RemoveContainer" containerID="fd70cdf408a1f20efa5010c8dded88a548b2f466f85653114cab719ae7a95fb8"
Jul 17 13:07:07.980789 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:07.980765    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg"]
Jul 17 13:07:07.988499 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:07.988476    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8svwg"]
Jul 17 13:07:09.634119 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:09.634082    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c211d417-f590-41fd-b593-28dcf4617240" path="/var/lib/kubelet/pods/c211d417-f590-41fd-b593-28dcf4617240/volumes"
Jul 17 13:07:13.399039 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:13.398932    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 13:07:13.415461 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:13.402466    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 13:07:14.474090 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474055    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"]
Jul 17 13:07:14.474462 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474371    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b" containerName="maas-api"
Jul 17 13:07:14.474462 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474382    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b" containerName="maas-api"
Jul 17 13:07:14.474462 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474391    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="180bb1a4-433a-4c64-bb7e-9b1e6321ddb3" containerName="revoke-keys"
Jul 17 13:07:14.474462 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474397    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="180bb1a4-433a-4c64-bb7e-9b1e6321ddb3" containerName="revoke-keys"
Jul 17 13:07:14.474462 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474408    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b" containerName="maas-api"
Jul 17 13:07:14.474462 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474414    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b" containerName="maas-api"
Jul 17 13:07:14.474462 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474428    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c211d417-f590-41fd-b593-28dcf4617240" containerName="revoke-keys"
Jul 17 13:07:14.474462 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474434    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c211d417-f590-41fd-b593-28dcf4617240" containerName="revoke-keys"
Jul 17 13:07:14.474462 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474440    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c211d417-f590-41fd-b593-28dcf4617240" containerName="revoke-keys"
Jul 17 13:07:14.474462 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474446    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c211d417-f590-41fd-b593-28dcf4617240" containerName="revoke-keys"
Jul 17 13:07:14.474462 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474451    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c211d417-f590-41fd-b593-28dcf4617240" containerName="revoke-keys"
Jul 17 13:07:14.474462 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474456    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c211d417-f590-41fd-b593-28dcf4617240" containerName="revoke-keys"
Jul 17 13:07:14.474884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474511    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c211d417-f590-41fd-b593-28dcf4617240" containerName="revoke-keys"
Jul 17 13:07:14.474884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474519    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b" containerName="maas-api"
Jul 17 13:07:14.474884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474524    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c211d417-f590-41fd-b593-28dcf4617240" containerName="revoke-keys"
Jul 17 13:07:14.474884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474529    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b" containerName="maas-api"
Jul 17 13:07:14.474884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474536    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="180bb1a4-433a-4c64-bb7e-9b1e6321ddb3" containerName="revoke-keys"
Jul 17 13:07:14.474884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474542    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c211d417-f590-41fd-b593-28dcf4617240" containerName="revoke-keys"
Jul 17 13:07:14.474884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474548    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b" containerName="maas-api"
Jul 17 13:07:14.474884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.474554    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b" containerName="maas-api"
Jul 17 13:07:14.477483 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.477460    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:07:14.482157 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.482137    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:07:14.483068 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.483045    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 13:07:14.483717 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.483699    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lwvzj\""
Jul 17 13:07:14.492874 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.492855    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-12f6c9d7-serving-cert\""
Jul 17 13:07:14.496704 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.496683    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"]
Jul 17 13:07:14.555048 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.555025    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcff1798-e8d8-4449-b8e3-eaf723b636c4-maas-api-tls\") pod \"maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5\" (UID: \"bcff1798-e8d8-4449-b8e3-eaf723b636c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:07:14.555138 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.555061    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcff1798-e8d8-4449-b8e3-eaf723b636c4-openshift-service-ca\") pod \"maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5\" (UID: \"bcff1798-e8d8-4449-b8e3-eaf723b636c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:07:14.555138 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.555086    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8x4z\" (UniqueName: \"kubernetes.io/projected/bcff1798-e8d8-4449-b8e3-eaf723b636c4-kube-api-access-z8x4z\") pod \"maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5\" (UID: \"bcff1798-e8d8-4449-b8e3-eaf723b636c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:07:14.655808 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.655780    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcff1798-e8d8-4449-b8e3-eaf723b636c4-maas-api-tls\") pod \"maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5\" (UID: \"bcff1798-e8d8-4449-b8e3-eaf723b636c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:07:14.655930 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.655828    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcff1798-e8d8-4449-b8e3-eaf723b636c4-openshift-service-ca\") pod \"maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5\" (UID: \"bcff1798-e8d8-4449-b8e3-eaf723b636c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:07:14.655930 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.655849    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8x4z\" (UniqueName: \"kubernetes.io/projected/bcff1798-e8d8-4449-b8e3-eaf723b636c4-kube-api-access-z8x4z\") pod \"maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5\" (UID: \"bcff1798-e8d8-4449-b8e3-eaf723b636c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:07:14.656038 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:14.655965    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-serving-cert: secret "maas-api-e2e-ait-12f6c9d7-serving-cert" not found
Jul 17 13:07:14.656038 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:14.656035    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcff1798-e8d8-4449-b8e3-eaf723b636c4-maas-api-tls podName:bcff1798-e8d8-4449-b8e3-eaf723b636c4 nodeName:}" failed. No retries permitted until 2026-07-17 13:07:15.15601627 +0000 UTC m=+2710.160063143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bcff1798-e8d8-4449-b8e3-eaf723b636c4-maas-api-tls") pod "maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" (UID: "bcff1798-e8d8-4449-b8e3-eaf723b636c4") : secret "maas-api-e2e-ait-12f6c9d7-serving-cert" not found
Jul 17 13:07:14.656571 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.656551    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcff1798-e8d8-4449-b8e3-eaf723b636c4-openshift-service-ca\") pod \"maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5\" (UID: \"bcff1798-e8d8-4449-b8e3-eaf723b636c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:07:14.670235 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:14.670207    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8x4z\" (UniqueName: \"kubernetes.io/projected/bcff1798-e8d8-4449-b8e3-eaf723b636c4-kube-api-access-z8x4z\") pod \"maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5\" (UID: \"bcff1798-e8d8-4449-b8e3-eaf723b636c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:07:15.159577 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:15.159546    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcff1798-e8d8-4449-b8e3-eaf723b636c4-maas-api-tls\") pod \"maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5\" (UID: \"bcff1798-e8d8-4449-b8e3-eaf723b636c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:07:15.161887 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:15.161869    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcff1798-e8d8-4449-b8e3-eaf723b636c4-maas-api-tls\") pod \"maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5\" (UID: \"bcff1798-e8d8-4449-b8e3-eaf723b636c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:07:15.388167 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:15.388134    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:07:15.515737 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:15.515678    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"]
Jul 17 13:07:15.517886 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:07:15.517861    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcff1798_e8d8_4449_b8e3_eaf723b636c4.slice/crio-c095d0c396f551c25e1baf2b90e5afb3097be597c78c16b0b364f496170b1d85 WatchSource:0}: Error finding container c095d0c396f551c25e1baf2b90e5afb3097be597c78c16b0b364f496170b1d85: Status 404 returned error can't find the container with id c095d0c396f551c25e1baf2b90e5afb3097be597c78c16b0b364f496170b1d85
Jul 17 13:07:15.997846 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:15.997799    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" event={"ID":"bcff1798-e8d8-4449-b8e3-eaf723b636c4","Type":"ContainerStarted","Data":"c095d0c396f551c25e1baf2b90e5afb3097be597c78c16b0b364f496170b1d85"}
Jul 17 13:07:16.573745 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:16.573678    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:07:17.005760 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:17.005711    2569 generic.go:356] "Generic (PLEG): container finished" podID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerID="e14fab895c3be4d522f1a1b148ce3d0e223035775b459bbb4e1747c1e310a5a8" exitCode=1
Jul 17 13:07:17.005922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:17.005777    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" event={"ID":"bcff1798-e8d8-4449-b8e3-eaf723b636c4","Type":"ContainerDied","Data":"e14fab895c3be4d522f1a1b148ce3d0e223035775b459bbb4e1747c1e310a5a8"}
Jul 17 13:07:17.006031 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:17.006016    2569 scope.go:117] "RemoveContainer" containerID="e14fab895c3be4d522f1a1b148ce3d0e223035775b459bbb4e1747c1e310a5a8"
Jul 17 13:07:18.012587 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:18.012555    2569 generic.go:356] "Generic (PLEG): container finished" podID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerID="486a4ee7557d160bdb235451b16b963efceb88576e09598eaf9f5879f02f766e" exitCode=1
Jul 17 13:07:18.013005 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:18.012611    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" event={"ID":"bcff1798-e8d8-4449-b8e3-eaf723b636c4","Type":"ContainerDied","Data":"486a4ee7557d160bdb235451b16b963efceb88576e09598eaf9f5879f02f766e"}
Jul 17 13:07:18.013005 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:18.012667    2569 scope.go:117] "RemoveContainer" containerID="e14fab895c3be4d522f1a1b148ce3d0e223035775b459bbb4e1747c1e310a5a8"
Jul 17 13:07:18.013005 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:18.012854    2569 scope.go:117] "RemoveContainer" containerID="486a4ee7557d160bdb235451b16b963efceb88576e09598eaf9f5879f02f766e"
Jul 17 13:07:18.013131 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:18.013068    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:07:19.019242 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:19.019212    2569 scope.go:117] "RemoveContainer" containerID="486a4ee7557d160bdb235451b16b963efceb88576e09598eaf9f5879f02f766e"
Jul 17 13:07:19.019607 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:19.019376    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:07:19.133960 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:19.133937    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx"]
Jul 17 13:07:19.134276 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:19.134263    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b" containerName="maas-api"
Jul 17 13:07:19.134325 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:19.134277    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b" containerName="maas-api"
Jul 17 13:07:19.137115 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:19.137098    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx"
Jul 17 13:07:19.139739 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:19.139717    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:07:19.145826 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:19.145806    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx"]
Jul 17 13:07:19.184001 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:19.183978    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc2b422-695b-498c-8111-657b0de0f2a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx\" (UID: \"2fc2b422-695b-498c-8111-657b0de0f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx"
Jul 17 13:07:19.284318 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:19.284262    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc2b422-695b-498c-8111-657b0de0f2a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx\" (UID: \"2fc2b422-695b-498c-8111-657b0de0f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx"
Jul 17 13:07:19.284852 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:19.284824    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc2b422-695b-498c-8111-657b0de0f2a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx\" (UID: \"2fc2b422-695b-498c-8111-657b0de0f2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx"
Jul 17 13:07:19.448700 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:19.448677    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx"
Jul 17 13:07:19.578453 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:19.578429    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx"]
Jul 17 13:07:19.580766 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:07:19.580721    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc2b422_695b_498c_8111_657b0de0f2a3.slice/crio-d543916dd0d3b9d179cdbc49b51d9273a76acb53d444b4e91598a4b576dc0b5a WatchSource:0}: Error finding container d543916dd0d3b9d179cdbc49b51d9273a76acb53d444b4e91598a4b576dc0b5a: Status 404 returned error can't find the container with id d543916dd0d3b9d179cdbc49b51d9273a76acb53d444b4e91598a4b576dc0b5a
Jul 17 13:07:20.025521 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:20.025486    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx" event={"ID":"2fc2b422-695b-498c-8111-657b0de0f2a3","Type":"ContainerStarted","Data":"4724e2fa19805aa54785d37309cd8e88feb8b64d2db2da69ed54cc65077cd324"}
Jul 17 13:07:20.025870 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:20.025547    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx" event={"ID":"2fc2b422-695b-498c-8111-657b0de0f2a3","Type":"ContainerStarted","Data":"d543916dd0d3b9d179cdbc49b51d9273a76acb53d444b4e91598a4b576dc0b5a"}
Jul 17 13:07:20.044377 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:20.044330    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx" podStartSLOduration=1.044315331 podStartE2EDuration="1.044315331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:07:20.043860704 +0000 UTC m=+2715.047907586" watchObservedRunningTime="2026-07-17 13:07:20.044315331 +0000 UTC m=+2715.048362213"
Jul 17 13:07:20.388381 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:20.388327    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:07:20.388670 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:20.388655    2569 scope.go:117] "RemoveContainer" containerID="486a4ee7557d160bdb235451b16b963efceb88576e09598eaf9f5879f02f766e"
Jul 17 13:07:20.388863 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:20.388842    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:07:21.031670 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:21.031642    2569 generic.go:356] "Generic (PLEG): container finished" podID="2fc2b422-695b-498c-8111-657b0de0f2a3" containerID="4724e2fa19805aa54785d37309cd8e88feb8b64d2db2da69ed54cc65077cd324" exitCode=7
Jul 17 13:07:21.032042 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:21.031677    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx" event={"ID":"2fc2b422-695b-498c-8111-657b0de0f2a3","Type":"ContainerDied","Data":"4724e2fa19805aa54785d37309cd8e88feb8b64d2db2da69ed54cc65077cd324"}
Jul 17 13:07:21.032042 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:21.031877    2569 scope.go:117] "RemoveContainer" containerID="4724e2fa19805aa54785d37309cd8e88feb8b64d2db2da69ed54cc65077cd324"
Jul 17 13:07:22.038488 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:22.038455    2569 generic.go:356] "Generic (PLEG): container finished" podID="2fc2b422-695b-498c-8111-657b0de0f2a3" containerID="154bf2d79f7c64fa1a3956ce3f78cc0540176ab5df309dac890bdcfdd304bf7c" exitCode=7
Jul 17 13:07:22.039136 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:22.038553    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx" event={"ID":"2fc2b422-695b-498c-8111-657b0de0f2a3","Type":"ContainerDied","Data":"154bf2d79f7c64fa1a3956ce3f78cc0540176ab5df309dac890bdcfdd304bf7c"}
Jul 17 13:07:22.039136 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:22.038630    2569 scope.go:117] "RemoveContainer" containerID="4724e2fa19805aa54785d37309cd8e88feb8b64d2db2da69ed54cc65077cd324"
Jul 17 13:07:22.039136 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:22.038696    2569 scope.go:117] "RemoveContainer" containerID="154bf2d79f7c64fa1a3956ce3f78cc0540176ab5df309dac890bdcfdd304bf7c"
Jul 17 13:07:22.039136 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:22.038918    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx_odh-ai-gateway-infra(2fc2b422-695b-498c-8111-657b0de0f2a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx" podUID="2fc2b422-695b-498c-8111-657b0de0f2a3"
Jul 17 13:07:23.044903 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:23.044880    2569 scope.go:117] "RemoveContainer" containerID="154bf2d79f7c64fa1a3956ce3f78cc0540176ab5df309dac890bdcfdd304bf7c"
Jul 17 13:07:23.045366 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:23.045113    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx_odh-ai-gateway-infra(2fc2b422-695b-498c-8111-657b0de0f2a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx" podUID="2fc2b422-695b-498c-8111-657b0de0f2a3"
Jul 17 13:07:25.389091 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:25.389066    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:07:25.389462 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:25.389447    2569 scope.go:117] "RemoveContainer" containerID="486a4ee7557d160bdb235451b16b963efceb88576e09598eaf9f5879f02f766e"
Jul 17 13:07:25.389623 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:25.389606    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:07:31.510620 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:31.510576    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn"]
Jul 17 13:07:31.511069 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:31.510959    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b" containerName="maas-api"
Jul 17 13:07:31.511069 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:31.510973    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e30576-e46b-4760-a491-9ddebcb0f49b" containerName="maas-api"
Jul 17 13:07:31.515397 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:31.515372    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn"
Jul 17 13:07:31.522311 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:31.522262    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn"]
Jul 17 13:07:31.572814 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:31.572782    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba82434c-ac58-4978-9d16-695cd01961b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn\" (UID: \"ba82434c-ac58-4978-9d16-695cd01961b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn"
Jul 17 13:07:31.673921 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:31.673898    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba82434c-ac58-4978-9d16-695cd01961b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn\" (UID: \"ba82434c-ac58-4978-9d16-695cd01961b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn"
Jul 17 13:07:31.674477 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:31.674457    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba82434c-ac58-4978-9d16-695cd01961b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn\" (UID: \"ba82434c-ac58-4978-9d16-695cd01961b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn"
Jul 17 13:07:31.828696 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:31.828626    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn"
Jul 17 13:07:31.958418 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:31.958392    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn"]
Jul 17 13:07:31.961788 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:07:31.961762    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba82434c_ac58_4978_9d16_695cd01961b7.slice/crio-209bd8dea71ffc56faddcaba3173cef75318067f98bfa2315a5dcc8b62064174 WatchSource:0}: Error finding container 209bd8dea71ffc56faddcaba3173cef75318067f98bfa2315a5dcc8b62064174: Status 404 returned error can't find the container with id 209bd8dea71ffc56faddcaba3173cef75318067f98bfa2315a5dcc8b62064174
Jul 17 13:07:32.088914 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:32.088858    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn" event={"ID":"ba82434c-ac58-4978-9d16-695cd01961b7","Type":"ContainerStarted","Data":"209bd8dea71ffc56faddcaba3173cef75318067f98bfa2315a5dcc8b62064174"}
Jul 17 13:07:33.098120 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:33.098083    2569 generic.go:356] "Generic (PLEG): container finished" podID="ba82434c-ac58-4978-9d16-695cd01961b7" containerID="adc903f58e72e8fd440183d3101f0d3d4463f9d104409e203eee5171c9aaa253" exitCode=6
Jul 17 13:07:33.098504 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:33.098147    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn" event={"ID":"ba82434c-ac58-4978-9d16-695cd01961b7","Type":"ContainerDied","Data":"adc903f58e72e8fd440183d3101f0d3d4463f9d104409e203eee5171c9aaa253"}
Jul 17 13:07:33.098504 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:33.098385    2569 scope.go:117] "RemoveContainer" containerID="adc903f58e72e8fd440183d3101f0d3d4463f9d104409e203eee5171c9aaa253"
Jul 17 13:07:34.104011 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:34.103951    2569 generic.go:356] "Generic (PLEG): container finished" podID="ba82434c-ac58-4978-9d16-695cd01961b7" containerID="fa766f745e41c1e8dd5925d3faf2e2348ffac54eb1b2e1aa71f38f6903e647fe" exitCode=6
Jul 17 13:07:34.104326 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:34.104039    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn" event={"ID":"ba82434c-ac58-4978-9d16-695cd01961b7","Type":"ContainerDied","Data":"fa766f745e41c1e8dd5925d3faf2e2348ffac54eb1b2e1aa71f38f6903e647fe"}
Jul 17 13:07:34.104326 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:34.104099    2569 scope.go:117] "RemoveContainer" containerID="adc903f58e72e8fd440183d3101f0d3d4463f9d104409e203eee5171c9aaa253"
Jul 17 13:07:34.104326 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:34.104263    2569 scope.go:117] "RemoveContainer" containerID="fa766f745e41c1e8dd5925d3faf2e2348ffac54eb1b2e1aa71f38f6903e647fe"
Jul 17 13:07:34.104466 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:34.104424    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn_odh-ai-gateway-infra(ba82434c-ac58-4978-9d16-695cd01961b7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn" podUID="ba82434c-ac58-4978-9d16-695cd01961b7"
Jul 17 13:07:35.112246 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:35.112218    2569 scope.go:117] "RemoveContainer" containerID="fa766f745e41c1e8dd5925d3faf2e2348ffac54eb1b2e1aa71f38f6903e647fe"
Jul 17 13:07:35.112663 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:35.112374    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn_odh-ai-gateway-infra(ba82434c-ac58-4978-9d16-695cd01961b7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn" podUID="ba82434c-ac58-4978-9d16-695cd01961b7"
Jul 17 13:07:37.629087 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:37.629046    2569 scope.go:117] "RemoveContainer" containerID="154bf2d79f7c64fa1a3956ce3f78cc0540176ab5df309dac890bdcfdd304bf7c"
Jul 17 13:07:38.128229 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:38.128202    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx" event={"ID":"2fc2b422-695b-498c-8111-657b0de0f2a3","Type":"ContainerStarted","Data":"1dac01c786cc2cd7e0c4b3143fadd82ca2312d5fbc70eb395311f0bc91501cf4"}
Jul 17 13:07:39.135403 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:39.135368    2569 generic.go:356] "Generic (PLEG): container finished" podID="2fc2b422-695b-498c-8111-657b0de0f2a3" containerID="1dac01c786cc2cd7e0c4b3143fadd82ca2312d5fbc70eb395311f0bc91501cf4" exitCode=7
Jul 17 13:07:39.135884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:39.135441    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx" event={"ID":"2fc2b422-695b-498c-8111-657b0de0f2a3","Type":"ContainerDied","Data":"1dac01c786cc2cd7e0c4b3143fadd82ca2312d5fbc70eb395311f0bc91501cf4"}
Jul 17 13:07:39.135884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:39.135567    2569 scope.go:117] "RemoveContainer" containerID="154bf2d79f7c64fa1a3956ce3f78cc0540176ab5df309dac890bdcfdd304bf7c"
Jul 17 13:07:39.136015 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:39.135900    2569 scope.go:117] "RemoveContainer" containerID="1dac01c786cc2cd7e0c4b3143fadd82ca2312d5fbc70eb395311f0bc91501cf4"
Jul 17 13:07:39.136108 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:39.136077    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx_odh-ai-gateway-infra(2fc2b422-695b-498c-8111-657b0de0f2a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx" podUID="2fc2b422-695b-498c-8111-657b0de0f2a3"
Jul 17 13:07:39.164779 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:39.164741    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx"]
Jul 17 13:07:39.628430 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:39.628408    2569 scope.go:117] "RemoveContainer" containerID="486a4ee7557d160bdb235451b16b963efceb88576e09598eaf9f5879f02f766e"
Jul 17 13:07:40.141711 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.141663    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" event={"ID":"bcff1798-e8d8-4449-b8e3-eaf723b636c4","Type":"ContainerStarted","Data":"02cd3d42dedcab651bf2c3746f0cc06c116863bf6602e2a90738d5733563bd36"}
Jul 17 13:07:40.142058 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.141930    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:07:40.162472 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.162427    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podStartSLOduration=24.885513336 podStartE2EDuration="26.162412426s" totalImagesPullingTime="1.27689909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:07:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:07:40.161369685 +0000 UTC m=+2735.165416569" watchObservedRunningTime="2026-07-17 13:07:40.162412426 +0000 UTC m=+2735.166459346"
Jul 17 13:07:40.275394 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.275370    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx"
Jul 17 13:07:40.328796 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.328771    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw"]
Jul 17 13:07:40.329174 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.329157    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fc2b422-695b-498c-8111-657b0de0f2a3" containerName="revoke-keys"
Jul 17 13:07:40.329229 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.329177    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fc2b422-695b-498c-8111-657b0de0f2a3" containerName="revoke-keys"
Jul 17 13:07:40.329229 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.329197    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fc2b422-695b-498c-8111-657b0de0f2a3" containerName="revoke-keys"
Jul 17 13:07:40.329229 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.329202    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fc2b422-695b-498c-8111-657b0de0f2a3" containerName="revoke-keys"
Jul 17 13:07:40.329331 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.329255    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fc2b422-695b-498c-8111-657b0de0f2a3" containerName="revoke-keys"
Jul 17 13:07:40.329331 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.329267    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fc2b422-695b-498c-8111-657b0de0f2a3" containerName="revoke-keys"
Jul 17 13:07:40.332163 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.332147    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw"
Jul 17 13:07:40.344357 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.344332    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw"]
Jul 17 13:07:40.347221 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.347202    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc2b422-695b-498c-8111-657b0de0f2a3-maas-api-service-ca\") pod \"2fc2b422-695b-498c-8111-657b0de0f2a3\" (UID: \"2fc2b422-695b-498c-8111-657b0de0f2a3\") "
Jul 17 13:07:40.347601 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.347576    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc2b422-695b-498c-8111-657b0de0f2a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2fc2b422-695b-498c-8111-657b0de0f2a3" (UID: "2fc2b422-695b-498c-8111-657b0de0f2a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:07:40.447973 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.447949    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e824c19-2d94-4ea7-a9c0-61c898e2ac15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw\" (UID: \"6e824c19-2d94-4ea7-a9c0-61c898e2ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw"
Jul 17 13:07:40.448068 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.448053    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc2b422-695b-498c-8111-657b0de0f2a3-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:07:40.549421 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.549362    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e824c19-2d94-4ea7-a9c0-61c898e2ac15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw\" (UID: \"6e824c19-2d94-4ea7-a9c0-61c898e2ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw"
Jul 17 13:07:40.549929 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.549912    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e824c19-2d94-4ea7-a9c0-61c898e2ac15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw\" (UID: \"6e824c19-2d94-4ea7-a9c0-61c898e2ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw"
Jul 17 13:07:40.641140 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.641119    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw"
Jul 17 13:07:40.767434 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:40.767365    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw"]
Jul 17 13:07:40.769548 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:07:40.769523    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e824c19_2d94_4ea7_a9c0_61c898e2ac15.slice/crio-b659516f6e57eebe18596c1a8d7de0ae4c1172b3ca273b74b6047084dea44fda WatchSource:0}: Error finding container b659516f6e57eebe18596c1a8d7de0ae4c1172b3ca273b74b6047084dea44fda: Status 404 returned error can't find the container with id b659516f6e57eebe18596c1a8d7de0ae4c1172b3ca273b74b6047084dea44fda
Jul 17 13:07:41.150058 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:41.149997    2569 generic.go:356] "Generic (PLEG): container finished" podID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerID="02cd3d42dedcab651bf2c3746f0cc06c116863bf6602e2a90738d5733563bd36" exitCode=1
Jul 17 13:07:41.150441 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:41.150075    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" event={"ID":"bcff1798-e8d8-4449-b8e3-eaf723b636c4","Type":"ContainerDied","Data":"02cd3d42dedcab651bf2c3746f0cc06c116863bf6602e2a90738d5733563bd36"}
Jul 17 13:07:41.150441 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:41.150131    2569 scope.go:117] "RemoveContainer" containerID="486a4ee7557d160bdb235451b16b963efceb88576e09598eaf9f5879f02f766e"
Jul 17 13:07:41.150441 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:41.150346    2569 scope.go:117] "RemoveContainer" containerID="02cd3d42dedcab651bf2c3746f0cc06c116863bf6602e2a90738d5733563bd36"
Jul 17 13:07:41.150599 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:41.150552    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:07:41.151788 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:41.151765    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw" event={"ID":"6e824c19-2d94-4ea7-a9c0-61c898e2ac15","Type":"ContainerStarted","Data":"acdcbf36cad2579bf443d5d5b87f471465f8995679b20f2c2ec33c4373be5080"}
Jul 17 13:07:41.151902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:41.151808    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw" event={"ID":"6e824c19-2d94-4ea7-a9c0-61c898e2ac15","Type":"ContainerStarted","Data":"b659516f6e57eebe18596c1a8d7de0ae4c1172b3ca273b74b6047084dea44fda"}
Jul 17 13:07:41.152941 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:41.152922    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx" event={"ID":"2fc2b422-695b-498c-8111-657b0de0f2a3","Type":"ContainerDied","Data":"d543916dd0d3b9d179cdbc49b51d9273a76acb53d444b4e91598a4b576dc0b5a"}
Jul 17 13:07:41.152941 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:41.152938    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx"
Jul 17 13:07:41.160626 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:41.160611    2569 scope.go:117] "RemoveContainer" containerID="1dac01c786cc2cd7e0c4b3143fadd82ca2312d5fbc70eb395311f0bc91501cf4"
Jul 17 13:07:41.188435 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:41.188398    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw" podStartSLOduration=1.188386267 podStartE2EDuration="1.188386267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:07:41.187915689 +0000 UTC m=+2736.191962572" watchObservedRunningTime="2026-07-17 13:07:41.188386267 +0000 UTC m=+2736.192433149"
Jul 17 13:07:41.201328 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:41.201309    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx"]
Jul 17 13:07:41.205722 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:41.205698    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-657jx"]
Jul 17 13:07:41.634120 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:41.634095    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fc2b422-695b-498c-8111-657b0de0f2a3" path="/var/lib/kubelet/pods/2fc2b422-695b-498c-8111-657b0de0f2a3/volumes"
Jul 17 13:07:42.159972 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:42.159949    2569 scope.go:117] "RemoveContainer" containerID="02cd3d42dedcab651bf2c3746f0cc06c116863bf6602e2a90738d5733563bd36"
Jul 17 13:07:42.160307 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:42.160122    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:07:42.161168 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:42.161145    2569 generic.go:356] "Generic (PLEG): container finished" podID="6e824c19-2d94-4ea7-a9c0-61c898e2ac15" containerID="acdcbf36cad2579bf443d5d5b87f471465f8995679b20f2c2ec33c4373be5080" exitCode=6
Jul 17 13:07:42.161230 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:42.161179    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw" event={"ID":"6e824c19-2d94-4ea7-a9c0-61c898e2ac15","Type":"ContainerDied","Data":"acdcbf36cad2579bf443d5d5b87f471465f8995679b20f2c2ec33c4373be5080"}
Jul 17 13:07:42.161347 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:42.161335    2569 scope.go:117] "RemoveContainer" containerID="acdcbf36cad2579bf443d5d5b87f471465f8995679b20f2c2ec33c4373be5080"
Jul 17 13:07:43.168149 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:43.168107    2569 generic.go:356] "Generic (PLEG): container finished" podID="6e824c19-2d94-4ea7-a9c0-61c898e2ac15" containerID="f00d9013a8027ed8973d10f03f1c62e34c934fe6d2b4840bd974459775cf7ab3" exitCode=6
Jul 17 13:07:43.168598 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:43.168186    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw" event={"ID":"6e824c19-2d94-4ea7-a9c0-61c898e2ac15","Type":"ContainerDied","Data":"f00d9013a8027ed8973d10f03f1c62e34c934fe6d2b4840bd974459775cf7ab3"}
Jul 17 13:07:43.168598 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:43.168282    2569 scope.go:117] "RemoveContainer" containerID="acdcbf36cad2579bf443d5d5b87f471465f8995679b20f2c2ec33c4373be5080"
Jul 17 13:07:43.168598 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:43.168495    2569 scope.go:117] "RemoveContainer" containerID="f00d9013a8027ed8973d10f03f1c62e34c934fe6d2b4840bd974459775cf7ab3"
Jul 17 13:07:43.168797 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:43.168731    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw_odh-ai-gateway-infra(6e824c19-2d94-4ea7-a9c0-61c898e2ac15)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw" podUID="6e824c19-2d94-4ea7-a9c0-61c898e2ac15"
Jul 17 13:07:44.174216 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:44.174188    2569 scope.go:117] "RemoveContainer" containerID="f00d9013a8027ed8973d10f03f1c62e34c934fe6d2b4840bd974459775cf7ab3"
Jul 17 13:07:44.174612 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:44.174352    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw_odh-ai-gateway-infra(6e824c19-2d94-4ea7-a9c0-61c898e2ac15)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw" podUID="6e824c19-2d94-4ea7-a9c0-61c898e2ac15"
Jul 17 13:07:45.388658 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:45.388577    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:07:45.389156 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:45.389073    2569 scope.go:117] "RemoveContainer" containerID="02cd3d42dedcab651bf2c3746f0cc06c116863bf6602e2a90738d5733563bd36"
Jul 17 13:07:45.389300 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:45.389275    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:07:47.628771 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:47.628725    2569 scope.go:117] "RemoveContainer" containerID="fa766f745e41c1e8dd5925d3faf2e2348ffac54eb1b2e1aa71f38f6903e647fe"
Jul 17 13:07:48.193823 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:48.193795    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn" event={"ID":"ba82434c-ac58-4978-9d16-695cd01961b7","Type":"ContainerStarted","Data":"4d6457daaf7851fe55696802929b0ba4cdd59767d36e47886596b72c2229b4e6"}
Jul 17 13:07:49.199931 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:49.199900    2569 generic.go:356] "Generic (PLEG): container finished" podID="ba82434c-ac58-4978-9d16-695cd01961b7" containerID="4d6457daaf7851fe55696802929b0ba4cdd59767d36e47886596b72c2229b4e6" exitCode=6
Jul 17 13:07:49.200357 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:49.199972    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn" event={"ID":"ba82434c-ac58-4978-9d16-695cd01961b7","Type":"ContainerDied","Data":"4d6457daaf7851fe55696802929b0ba4cdd59767d36e47886596b72c2229b4e6"}
Jul 17 13:07:49.200357 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:49.200035    2569 scope.go:117] "RemoveContainer" containerID="fa766f745e41c1e8dd5925d3faf2e2348ffac54eb1b2e1aa71f38f6903e647fe"
Jul 17 13:07:49.200357 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:49.200320    2569 scope.go:117] "RemoveContainer" containerID="4d6457daaf7851fe55696802929b0ba4cdd59767d36e47886596b72c2229b4e6"
Jul 17 13:07:49.200566 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:49.200540    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn_odh-ai-gateway-infra(ba82434c-ac58-4978-9d16-695cd01961b7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn" podUID="ba82434c-ac58-4978-9d16-695cd01961b7"
Jul 17 13:07:49.228828 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:49.228802    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn"]
Jul 17 13:07:50.339940 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:50.339917    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn"
Jul 17 13:07:50.425975 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:50.425948    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba82434c-ac58-4978-9d16-695cd01961b7-maas-api-service-ca\") pod \"ba82434c-ac58-4978-9d16-695cd01961b7\" (UID: \"ba82434c-ac58-4978-9d16-695cd01961b7\") "
Jul 17 13:07:50.426263 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:50.426241    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba82434c-ac58-4978-9d16-695cd01961b7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba82434c-ac58-4978-9d16-695cd01961b7" (UID: "ba82434c-ac58-4978-9d16-695cd01961b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:07:50.526478 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:50.526458    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba82434c-ac58-4978-9d16-695cd01961b7-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:07:51.213691 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:51.213666    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn"
Jul 17 13:07:51.213860 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:51.213712    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn" event={"ID":"ba82434c-ac58-4978-9d16-695cd01961b7","Type":"ContainerDied","Data":"209bd8dea71ffc56faddcaba3173cef75318067f98bfa2315a5dcc8b62064174"}
Jul 17 13:07:51.213860 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:51.213804    2569 scope.go:117] "RemoveContainer" containerID="4d6457daaf7851fe55696802929b0ba4cdd59767d36e47886596b72c2229b4e6"
Jul 17 13:07:51.237650 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:51.237608    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn"]
Jul 17 13:07:51.244416 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:51.244392    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qqxzn"]
Jul 17 13:07:51.633189 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:51.633165    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba82434c-ac58-4978-9d16-695cd01961b7" path="/var/lib/kubelet/pods/ba82434c-ac58-4978-9d16-695cd01961b7/volumes"
Jul 17 13:07:55.631892 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:55.631859    2569 scope.go:117] "RemoveContainer" containerID="f00d9013a8027ed8973d10f03f1c62e34c934fe6d2b4840bd974459775cf7ab3"
Jul 17 13:07:56.241339 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:56.241313    2569 generic.go:356] "Generic (PLEG): container finished" podID="6e824c19-2d94-4ea7-a9c0-61c898e2ac15" containerID="d39d0e6176f96a3e4c79151b1b65bd3ece9037f6d132d7d7c4aa8a48511cc870" exitCode=6
Jul 17 13:07:56.241446 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:56.241378    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw" event={"ID":"6e824c19-2d94-4ea7-a9c0-61c898e2ac15","Type":"ContainerDied","Data":"d39d0e6176f96a3e4c79151b1b65bd3ece9037f6d132d7d7c4aa8a48511cc870"}
Jul 17 13:07:56.241446 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:56.241438    2569 scope.go:117] "RemoveContainer" containerID="f00d9013a8027ed8973d10f03f1c62e34c934fe6d2b4840bd974459775cf7ab3"
Jul 17 13:07:56.241706 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:56.241689    2569 scope.go:117] "RemoveContainer" containerID="d39d0e6176f96a3e4c79151b1b65bd3ece9037f6d132d7d7c4aa8a48511cc870"
Jul 17 13:07:56.241948 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:07:56.241928    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw_odh-ai-gateway-infra(6e824c19-2d94-4ea7-a9c0-61c898e2ac15)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw" podUID="6e824c19-2d94-4ea7-a9c0-61c898e2ac15"
Jul 17 13:07:57.268250 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:57.268213    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw"]
Jul 17 13:07:57.403955 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:57.403930    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw"
Jul 17 13:07:57.479942 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:57.479915    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e824c19-2d94-4ea7-a9c0-61c898e2ac15-maas-api-service-ca\") pod \"6e824c19-2d94-4ea7-a9c0-61c898e2ac15\" (UID: \"6e824c19-2d94-4ea7-a9c0-61c898e2ac15\") "
Jul 17 13:07:57.480241 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:57.480221    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e824c19-2d94-4ea7-a9c0-61c898e2ac15-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6e824c19-2d94-4ea7-a9c0-61c898e2ac15" (UID: "6e824c19-2d94-4ea7-a9c0-61c898e2ac15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:07:57.580853 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:57.580786    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e824c19-2d94-4ea7-a9c0-61c898e2ac15-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:07:58.254320 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:58.254287    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw" event={"ID":"6e824c19-2d94-4ea7-a9c0-61c898e2ac15","Type":"ContainerDied","Data":"b659516f6e57eebe18596c1a8d7de0ae4c1172b3ca273b74b6047084dea44fda"}
Jul 17 13:07:58.254320 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:58.254320    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw"
Jul 17 13:07:58.254511 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:58.254343    2569 scope.go:117] "RemoveContainer" containerID="d39d0e6176f96a3e4c79151b1b65bd3ece9037f6d132d7d7c4aa8a48511cc870"
Jul 17 13:07:58.272822 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:58.272797    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw"]
Jul 17 13:07:58.277764 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:58.277728    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-p6grw"]
Jul 17 13:07:59.637391 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:07:59.637357    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e824c19-2d94-4ea7-a9c0-61c898e2ac15" path="/var/lib/kubelet/pods/6e824c19-2d94-4ea7-a9c0-61c898e2ac15/volumes"
Jul 17 13:08:00.629106 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:00.629079    2569 scope.go:117] "RemoveContainer" containerID="02cd3d42dedcab651bf2c3746f0cc06c116863bf6602e2a90738d5733563bd36"
Jul 17 13:08:00.629379 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:00.629364    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 13:08:01.270131 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:01.270096    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" event={"ID":"bcff1798-e8d8-4449-b8e3-eaf723b636c4","Type":"ContainerStarted","Data":"74bd6dc948892d3cf439100d602ea4a614dc5864856ae597419499df79fa9001"}
Jul 17 13:08:01.270453 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:01.270295    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:08:02.277320 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:02.277286    2569 generic.go:356] "Generic (PLEG): container finished" podID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerID="74bd6dc948892d3cf439100d602ea4a614dc5864856ae597419499df79fa9001" exitCode=1
Jul 17 13:08:02.277768 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:02.277346    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" event={"ID":"bcff1798-e8d8-4449-b8e3-eaf723b636c4","Type":"ContainerDied","Data":"74bd6dc948892d3cf439100d602ea4a614dc5864856ae597419499df79fa9001"}
Jul 17 13:08:02.277768 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:02.277404    2569 scope.go:117] "RemoveContainer" containerID="02cd3d42dedcab651bf2c3746f0cc06c116863bf6602e2a90738d5733563bd36"
Jul 17 13:08:02.277768 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:02.277564    2569 scope.go:117] "RemoveContainer" containerID="74bd6dc948892d3cf439100d602ea4a614dc5864856ae597419499df79fa9001"
Jul 17 13:08:02.277912 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:08:02.277777    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:08:03.284606 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:03.284581    2569 scope.go:117] "RemoveContainer" containerID="74bd6dc948892d3cf439100d602ea4a614dc5864856ae597419499df79fa9001"
Jul 17 13:08:03.285049 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:08:03.284746    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:08:05.388952 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:05.388908    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:08:05.389461 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:05.389374    2569 scope.go:117] "RemoveContainer" containerID="74bd6dc948892d3cf439100d602ea4a614dc5864856ae597419499df79fa9001"
Jul 17 13:08:05.389607 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:08:05.389582    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:08:19.193536 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.193497    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw"]
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.193878    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e824c19-2d94-4ea7-a9c0-61c898e2ac15" containerName="revoke-keys"
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.193893    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e824c19-2d94-4ea7-a9c0-61c898e2ac15" containerName="revoke-keys"
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.193900    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba82434c-ac58-4978-9d16-695cd01961b7" containerName="revoke-keys"
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.193906    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba82434c-ac58-4978-9d16-695cd01961b7" containerName="revoke-keys"
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.193916    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fc2b422-695b-498c-8111-657b0de0f2a3" containerName="revoke-keys"
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.193921    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fc2b422-695b-498c-8111-657b0de0f2a3" containerName="revoke-keys"
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.193930    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba82434c-ac58-4978-9d16-695cd01961b7" containerName="revoke-keys"
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.193936    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba82434c-ac58-4978-9d16-695cd01961b7" containerName="revoke-keys"
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.193946    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e824c19-2d94-4ea7-a9c0-61c898e2ac15" containerName="revoke-keys"
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.193953    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e824c19-2d94-4ea7-a9c0-61c898e2ac15" containerName="revoke-keys"
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.193960    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e824c19-2d94-4ea7-a9c0-61c898e2ac15" containerName="revoke-keys"
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.193965    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e824c19-2d94-4ea7-a9c0-61c898e2ac15" containerName="revoke-keys"
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.194047    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e824c19-2d94-4ea7-a9c0-61c898e2ac15" containerName="revoke-keys"
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.194054    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e824c19-2d94-4ea7-a9c0-61c898e2ac15" containerName="revoke-keys"
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.194061    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba82434c-ac58-4978-9d16-695cd01961b7" containerName="revoke-keys"
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.194068    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fc2b422-695b-498c-8111-657b0de0f2a3" containerName="revoke-keys"
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.194074    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba82434c-ac58-4978-9d16-695cd01961b7" containerName="revoke-keys"
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.194080    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e824c19-2d94-4ea7-a9c0-61c898e2ac15" containerName="revoke-keys"
Jul 17 13:08:19.194116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.194085    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba82434c-ac58-4978-9d16-695cd01961b7" containerName="revoke-keys"
Jul 17 13:08:19.197254 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.197231    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw"
Jul 17 13:08:19.200062 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.200033    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:08:19.205533 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.205274    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw"]
Jul 17 13:08:19.348658 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.348628    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357966fe-ee46-4298-8c66-3871c8e61007-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw\" (UID: \"357966fe-ee46-4298-8c66-3871c8e61007\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw"
Jul 17 13:08:19.449599 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.449533    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357966fe-ee46-4298-8c66-3871c8e61007-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw\" (UID: \"357966fe-ee46-4298-8c66-3871c8e61007\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw"
Jul 17 13:08:19.450145 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.450123    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357966fe-ee46-4298-8c66-3871c8e61007-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw\" (UID: \"357966fe-ee46-4298-8c66-3871c8e61007\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw"
Jul 17 13:08:19.510657 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.510636    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw"
Jul 17 13:08:19.629381 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.629353    2569 scope.go:117] "RemoveContainer" containerID="74bd6dc948892d3cf439100d602ea4a614dc5864856ae597419499df79fa9001"
Jul 17 13:08:19.629591 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:08:19.629567    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:08:19.848652 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:19.848631    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw"]
Jul 17 13:08:19.851402 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:08:19.851368    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357966fe_ee46_4298_8c66_3871c8e61007.slice/crio-53c4684245a8a494f2d4218f19746b3803deda64e28d7ba8b3442b9f77302076 WatchSource:0}: Error finding container 53c4684245a8a494f2d4218f19746b3803deda64e28d7ba8b3442b9f77302076: Status 404 returned error can't find the container with id 53c4684245a8a494f2d4218f19746b3803deda64e28d7ba8b3442b9f77302076
Jul 17 13:08:20.363588 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:20.363555    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw" event={"ID":"357966fe-ee46-4298-8c66-3871c8e61007","Type":"ContainerStarted","Data":"9d235267c0662e02f65fb1c20e35713823b69e10882a288ba99d96ed042dfc50"}
Jul 17 13:08:20.363912 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:20.363615    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw" event={"ID":"357966fe-ee46-4298-8c66-3871c8e61007","Type":"ContainerStarted","Data":"53c4684245a8a494f2d4218f19746b3803deda64e28d7ba8b3442b9f77302076"}
Jul 17 13:08:20.383824 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:20.383781    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw" podStartSLOduration=1.383762948 podStartE2EDuration="1.383762948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:08:20.382093347 +0000 UTC m=+2775.386140232" watchObservedRunningTime="2026-07-17 13:08:20.383762948 +0000 UTC m=+2775.387809831"
Jul 17 13:08:21.370198 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:21.370164    2569 generic.go:356] "Generic (PLEG): container finished" podID="357966fe-ee46-4298-8c66-3871c8e61007" containerID="9d235267c0662e02f65fb1c20e35713823b69e10882a288ba99d96ed042dfc50" exitCode=7
Jul 17 13:08:21.370643 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:21.370260    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw" event={"ID":"357966fe-ee46-4298-8c66-3871c8e61007","Type":"ContainerDied","Data":"9d235267c0662e02f65fb1c20e35713823b69e10882a288ba99d96ed042dfc50"}
Jul 17 13:08:21.370643 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:21.370442    2569 scope.go:117] "RemoveContainer" containerID="9d235267c0662e02f65fb1c20e35713823b69e10882a288ba99d96ed042dfc50"
Jul 17 13:08:22.377853 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:22.377818    2569 generic.go:356] "Generic (PLEG): container finished" podID="357966fe-ee46-4298-8c66-3871c8e61007" containerID="9c2ee1a31a617502a73d99b0e873b8ab4e80c3125bace0399edcda0821341b54" exitCode=7
Jul 17 13:08:22.378260 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:22.377906    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw" event={"ID":"357966fe-ee46-4298-8c66-3871c8e61007","Type":"ContainerDied","Data":"9c2ee1a31a617502a73d99b0e873b8ab4e80c3125bace0399edcda0821341b54"}
Jul 17 13:08:22.378260 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:22.377971    2569 scope.go:117] "RemoveContainer" containerID="9d235267c0662e02f65fb1c20e35713823b69e10882a288ba99d96ed042dfc50"
Jul 17 13:08:22.378260 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:22.378241    2569 scope.go:117] "RemoveContainer" containerID="9c2ee1a31a617502a73d99b0e873b8ab4e80c3125bace0399edcda0821341b54"
Jul 17 13:08:22.378498 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:08:22.378475    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw_odh-ai-gateway-infra(357966fe-ee46-4298-8c66-3871c8e61007)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw" podUID="357966fe-ee46-4298-8c66-3871c8e61007"
Jul 17 13:08:23.385008 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:23.384979    2569 scope.go:117] "RemoveContainer" containerID="9c2ee1a31a617502a73d99b0e873b8ab4e80c3125bace0399edcda0821341b54"
Jul 17 13:08:23.385378 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:08:23.385157    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw_odh-ai-gateway-infra(357966fe-ee46-4298-8c66-3871c8e61007)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw" podUID="357966fe-ee46-4298-8c66-3871c8e61007"
Jul 17 13:08:30.380633 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:30.380428    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p"]
Jul 17 13:08:30.381948 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:30.381918    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba82434c-ac58-4978-9d16-695cd01961b7" containerName="revoke-keys"
Jul 17 13:08:30.382126 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:30.382113    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba82434c-ac58-4978-9d16-695cd01961b7" containerName="revoke-keys"
Jul 17 13:08:30.387339 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:30.387316    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p"
Jul 17 13:08:30.390812 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:30.390718    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p"]
Jul 17 13:08:30.429450 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:30.429427    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43eec4b7-5f87-431a-96dc-4dbcf2e54a4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p\" (UID: \"43eec4b7-5f87-431a-96dc-4dbcf2e54a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p"
Jul 17 13:08:30.530284 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:30.530260    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43eec4b7-5f87-431a-96dc-4dbcf2e54a4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p\" (UID: \"43eec4b7-5f87-431a-96dc-4dbcf2e54a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p"
Jul 17 13:08:30.530847 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:30.530830    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43eec4b7-5f87-431a-96dc-4dbcf2e54a4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p\" (UID: \"43eec4b7-5f87-431a-96dc-4dbcf2e54a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p"
Jul 17 13:08:30.699643 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:30.699572    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p"
Jul 17 13:08:30.826969 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:30.826945    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p"]
Jul 17 13:08:30.828858 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:08:30.828818    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43eec4b7_5f87_431a_96dc_4dbcf2e54a4c.slice/crio-3c98b4acd58d4a81af8d7bec54357a32f7e9e9f5f09412124b6c00b29aaddaad WatchSource:0}: Error finding container 3c98b4acd58d4a81af8d7bec54357a32f7e9e9f5f09412124b6c00b29aaddaad: Status 404 returned error can't find the container with id 3c98b4acd58d4a81af8d7bec54357a32f7e9e9f5f09412124b6c00b29aaddaad
Jul 17 13:08:31.427242 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:31.427209    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p" event={"ID":"43eec4b7-5f87-431a-96dc-4dbcf2e54a4c","Type":"ContainerStarted","Data":"e6613c1dd1457847cb00e3a5b5329b0a0578c07d70b2683b606692966205e20c"}
Jul 17 13:08:31.427500 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:31.427258    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p" event={"ID":"43eec4b7-5f87-431a-96dc-4dbcf2e54a4c","Type":"ContainerStarted","Data":"3c98b4acd58d4a81af8d7bec54357a32f7e9e9f5f09412124b6c00b29aaddaad"}
Jul 17 13:08:31.446971 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:31.446914    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p" podStartSLOduration=1.4468958029999999 podStartE2EDuration="1.446895803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:08:31.44525476 +0000 UTC m=+2786.449301645" watchObservedRunningTime="2026-07-17 13:08:31.446895803 +0000 UTC m=+2786.450942686"
Jul 17 13:08:32.434918 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:32.434887    2569 generic.go:356] "Generic (PLEG): container finished" podID="43eec4b7-5f87-431a-96dc-4dbcf2e54a4c" containerID="e6613c1dd1457847cb00e3a5b5329b0a0578c07d70b2683b606692966205e20c" exitCode=6
Jul 17 13:08:32.435295 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:32.434977    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p" event={"ID":"43eec4b7-5f87-431a-96dc-4dbcf2e54a4c","Type":"ContainerDied","Data":"e6613c1dd1457847cb00e3a5b5329b0a0578c07d70b2683b606692966205e20c"}
Jul 17 13:08:32.435295 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:32.435147    2569 scope.go:117] "RemoveContainer" containerID="e6613c1dd1457847cb00e3a5b5329b0a0578c07d70b2683b606692966205e20c"
Jul 17 13:08:33.441806 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:33.441720    2569 generic.go:356] "Generic (PLEG): container finished" podID="43eec4b7-5f87-431a-96dc-4dbcf2e54a4c" containerID="c4c3b56048ff0ba9be41489103be8a89075edcb471a3a632bae699d8d0784bcc" exitCode=6
Jul 17 13:08:33.442166 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:33.441801    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p" event={"ID":"43eec4b7-5f87-431a-96dc-4dbcf2e54a4c","Type":"ContainerDied","Data":"c4c3b56048ff0ba9be41489103be8a89075edcb471a3a632bae699d8d0784bcc"}
Jul 17 13:08:33.442166 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:33.441878    2569 scope.go:117] "RemoveContainer" containerID="e6613c1dd1457847cb00e3a5b5329b0a0578c07d70b2683b606692966205e20c"
Jul 17 13:08:33.442166 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:33.442106    2569 scope.go:117] "RemoveContainer" containerID="c4c3b56048ff0ba9be41489103be8a89075edcb471a3a632bae699d8d0784bcc"
Jul 17 13:08:33.442341 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:08:33.442319    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p_odh-ai-gateway-infra(43eec4b7-5f87-431a-96dc-4dbcf2e54a4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p" podUID="43eec4b7-5f87-431a-96dc-4dbcf2e54a4c"
Jul 17 13:08:33.629080 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:33.629060    2569 scope.go:117] "RemoveContainer" containerID="74bd6dc948892d3cf439100d602ea4a614dc5864856ae597419499df79fa9001"
Jul 17 13:08:33.629244 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:08:33.629225    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:08:34.456265 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:34.456236    2569 scope.go:117] "RemoveContainer" containerID="c4c3b56048ff0ba9be41489103be8a89075edcb471a3a632bae699d8d0784bcc"
Jul 17 13:08:34.456669 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:08:34.456424    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p_odh-ai-gateway-infra(43eec4b7-5f87-431a-96dc-4dbcf2e54a4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p" podUID="43eec4b7-5f87-431a-96dc-4dbcf2e54a4c"
Jul 17 13:08:37.634260 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:37.634224    2569 scope.go:117] "RemoveContainer" containerID="9c2ee1a31a617502a73d99b0e873b8ab4e80c3125bace0399edcda0821341b54"
Jul 17 13:08:38.477272 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:38.477246    2569 generic.go:356] "Generic (PLEG): container finished" podID="357966fe-ee46-4298-8c66-3871c8e61007" containerID="7b7f2ca169fd4e859d60411394f728828a9e58e57d0351a63cba8b47da99060d" exitCode=7
Jul 17 13:08:38.477396 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:38.477316    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw" event={"ID":"357966fe-ee46-4298-8c66-3871c8e61007","Type":"ContainerDied","Data":"7b7f2ca169fd4e859d60411394f728828a9e58e57d0351a63cba8b47da99060d"}
Jul 17 13:08:38.477396 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:38.477379    2569 scope.go:117] "RemoveContainer" containerID="9c2ee1a31a617502a73d99b0e873b8ab4e80c3125bace0399edcda0821341b54"
Jul 17 13:08:38.477642 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:38.477621    2569 scope.go:117] "RemoveContainer" containerID="7b7f2ca169fd4e859d60411394f728828a9e58e57d0351a63cba8b47da99060d"
Jul 17 13:08:38.477855 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:08:38.477832    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw_odh-ai-gateway-infra(357966fe-ee46-4298-8c66-3871c8e61007)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw" podUID="357966fe-ee46-4298-8c66-3871c8e61007"
Jul 17 13:08:39.505380 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:39.505351    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw"]
Jul 17 13:08:39.637556 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:39.637533    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw"
Jul 17 13:08:39.696149 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:39.696128    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357966fe-ee46-4298-8c66-3871c8e61007-maas-api-service-ca\") pod \"357966fe-ee46-4298-8c66-3871c8e61007\" (UID: \"357966fe-ee46-4298-8c66-3871c8e61007\") "
Jul 17 13:08:39.696442 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:39.696420    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357966fe-ee46-4298-8c66-3871c8e61007-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "357966fe-ee46-4298-8c66-3871c8e61007" (UID: "357966fe-ee46-4298-8c66-3871c8e61007"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:08:39.797655 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:39.797596    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357966fe-ee46-4298-8c66-3871c8e61007-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:08:40.489188 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:40.489159    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw"
Jul 17 13:08:40.489349 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:40.489183    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw" event={"ID":"357966fe-ee46-4298-8c66-3871c8e61007","Type":"ContainerDied","Data":"53c4684245a8a494f2d4218f19746b3803deda64e28d7ba8b3442b9f77302076"}
Jul 17 13:08:40.489349 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:40.489267    2569 scope.go:117] "RemoveContainer" containerID="7b7f2ca169fd4e859d60411394f728828a9e58e57d0351a63cba8b47da99060d"
Jul 17 13:08:40.512172 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:40.512146    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw"]
Jul 17 13:08:40.518203 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:40.518182    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-462mw"]
Jul 17 13:08:41.633890 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:41.633862    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="357966fe-ee46-4298-8c66-3871c8e61007" path="/var/lib/kubelet/pods/357966fe-ee46-4298-8c66-3871c8e61007/volumes"
Jul 17 13:08:45.630372 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:45.630336    2569 scope.go:117] "RemoveContainer" containerID="c4c3b56048ff0ba9be41489103be8a89075edcb471a3a632bae699d8d0784bcc"
Jul 17 13:08:45.630825 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:45.630373    2569 scope.go:117] "RemoveContainer" containerID="74bd6dc948892d3cf439100d602ea4a614dc5864856ae597419499df79fa9001"
Jul 17 13:08:46.519586 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:46.519531    2569 generic.go:356] "Generic (PLEG): container finished" podID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerID="3d3bd53ccc87d38e5ac00218c2e91639dd179589f67a763b0ddca1d8324d4ae1" exitCode=1
Jul 17 13:08:46.519683 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:46.519603    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" event={"ID":"bcff1798-e8d8-4449-b8e3-eaf723b636c4","Type":"ContainerDied","Data":"3d3bd53ccc87d38e5ac00218c2e91639dd179589f67a763b0ddca1d8324d4ae1"}
Jul 17 13:08:46.519683 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:46.519664    2569 scope.go:117] "RemoveContainer" containerID="74bd6dc948892d3cf439100d602ea4a614dc5864856ae597419499df79fa9001"
Jul 17 13:08:46.520015 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:46.519992    2569 scope.go:117] "RemoveContainer" containerID="3d3bd53ccc87d38e5ac00218c2e91639dd179589f67a763b0ddca1d8324d4ae1"
Jul 17 13:08:46.520218 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:08:46.520194    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:08:46.521524 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:46.521499    2569 generic.go:356] "Generic (PLEG): container finished" podID="43eec4b7-5f87-431a-96dc-4dbcf2e54a4c" containerID="10396489e1136146c766f49578414d2addc40565e65f727710c266a2329cde32" exitCode=6
Jul 17 13:08:46.521614 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:46.521572    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p" event={"ID":"43eec4b7-5f87-431a-96dc-4dbcf2e54a4c","Type":"ContainerDied","Data":"10396489e1136146c766f49578414d2addc40565e65f727710c266a2329cde32"}
Jul 17 13:08:46.521845 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:46.521832    2569 scope.go:117] "RemoveContainer" containerID="10396489e1136146c766f49578414d2addc40565e65f727710c266a2329cde32"
Jul 17 13:08:46.521978 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:08:46.521964    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p_odh-ai-gateway-infra(43eec4b7-5f87-431a-96dc-4dbcf2e54a4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p" podUID="43eec4b7-5f87-431a-96dc-4dbcf2e54a4c"
Jul 17 13:08:46.529708 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:46.529683    2569 scope.go:117] "RemoveContainer" containerID="c4c3b56048ff0ba9be41489103be8a89075edcb471a3a632bae699d8d0784bcc"
Jul 17 13:08:47.588885 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:47.588844    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p"]
Jul 17 13:08:47.716117 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:47.716093    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p"
Jul 17 13:08:47.756196 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:47.756168    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43eec4b7-5f87-431a-96dc-4dbcf2e54a4c-maas-api-service-ca\") pod \"43eec4b7-5f87-431a-96dc-4dbcf2e54a4c\" (UID: \"43eec4b7-5f87-431a-96dc-4dbcf2e54a4c\") "
Jul 17 13:08:47.756480 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:47.756458    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43eec4b7-5f87-431a-96dc-4dbcf2e54a4c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43eec4b7-5f87-431a-96dc-4dbcf2e54a4c" (UID: "43eec4b7-5f87-431a-96dc-4dbcf2e54a4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:08:47.857799 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:47.857729    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43eec4b7-5f87-431a-96dc-4dbcf2e54a4c-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:08:48.535975 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:48.535950    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p"
Jul 17 13:08:48.536101 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:48.535971    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p" event={"ID":"43eec4b7-5f87-431a-96dc-4dbcf2e54a4c","Type":"ContainerDied","Data":"3c98b4acd58d4a81af8d7bec54357a32f7e9e9f5f09412124b6c00b29aaddaad"}
Jul 17 13:08:48.536101 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:48.536034    2569 scope.go:117] "RemoveContainer" containerID="10396489e1136146c766f49578414d2addc40565e65f727710c266a2329cde32"
Jul 17 13:08:48.562372 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:48.562344    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p"]
Jul 17 13:08:48.565941 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:48.565917    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rg78p"]
Jul 17 13:08:49.633469 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:49.633440    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43eec4b7-5f87-431a-96dc-4dbcf2e54a4c" path="/var/lib/kubelet/pods/43eec4b7-5f87-431a-96dc-4dbcf2e54a4c/volumes"
Jul 17 13:08:51.270899 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:51.270874    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:08:51.271383 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:51.271189    2569 scope.go:117] "RemoveContainer" containerID="3d3bd53ccc87d38e5ac00218c2e91639dd179589f67a763b0ddca1d8324d4ae1"
Jul 17 13:08:51.271383 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:08:51.271350    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:08:55.388731 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:55.388692    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:08:55.389258 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:08:55.389168    2569 scope.go:117] "RemoveContainer" containerID="3d3bd53ccc87d38e5ac00218c2e91639dd179589f67a763b0ddca1d8324d4ae1"
Jul 17 13:08:55.389396 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:08:55.389373    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:09:09.628991 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:09.628951    2569 scope.go:117] "RemoveContainer" containerID="3d3bd53ccc87d38e5ac00218c2e91639dd179589f67a763b0ddca1d8324d4ae1"
Jul 17 13:09:09.629445 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:09:09.629172    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:09:19.253391 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.253348    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np"]
Jul 17 13:09:19.253937 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.253918    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43eec4b7-5f87-431a-96dc-4dbcf2e54a4c" containerName="revoke-keys"
Jul 17 13:09:19.254020 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.253943    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="43eec4b7-5f87-431a-96dc-4dbcf2e54a4c" containerName="revoke-keys"
Jul 17 13:09:19.254020 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.253970    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43eec4b7-5f87-431a-96dc-4dbcf2e54a4c" containerName="revoke-keys"
Jul 17 13:09:19.254020 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.253979    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="43eec4b7-5f87-431a-96dc-4dbcf2e54a4c" containerName="revoke-keys"
Jul 17 13:09:19.254020 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.253990    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357966fe-ee46-4298-8c66-3871c8e61007" containerName="revoke-keys"
Jul 17 13:09:19.254020 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.253999    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="357966fe-ee46-4298-8c66-3871c8e61007" containerName="revoke-keys"
Jul 17 13:09:19.254268 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.254010    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43eec4b7-5f87-431a-96dc-4dbcf2e54a4c" containerName="revoke-keys"
Jul 17 13:09:19.254268 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.254031    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="43eec4b7-5f87-431a-96dc-4dbcf2e54a4c" containerName="revoke-keys"
Jul 17 13:09:19.254268 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.254049    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357966fe-ee46-4298-8c66-3871c8e61007" containerName="revoke-keys"
Jul 17 13:09:19.254268 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.254059    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="357966fe-ee46-4298-8c66-3871c8e61007" containerName="revoke-keys"
Jul 17 13:09:19.254268 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.254071    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357966fe-ee46-4298-8c66-3871c8e61007" containerName="revoke-keys"
Jul 17 13:09:19.254268 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.254080    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="357966fe-ee46-4298-8c66-3871c8e61007" containerName="revoke-keys"
Jul 17 13:09:19.254268 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.254181    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="43eec4b7-5f87-431a-96dc-4dbcf2e54a4c" containerName="revoke-keys"
Jul 17 13:09:19.254268 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.254193    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="357966fe-ee46-4298-8c66-3871c8e61007" containerName="revoke-keys"
Jul 17 13:09:19.254268 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.254210    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="357966fe-ee46-4298-8c66-3871c8e61007" containerName="revoke-keys"
Jul 17 13:09:19.254268 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.254220    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="43eec4b7-5f87-431a-96dc-4dbcf2e54a4c" containerName="revoke-keys"
Jul 17 13:09:19.254268 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.254229    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="43eec4b7-5f87-431a-96dc-4dbcf2e54a4c" containerName="revoke-keys"
Jul 17 13:09:19.254268 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.254239    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="357966fe-ee46-4298-8c66-3871c8e61007" containerName="revoke-keys"
Jul 17 13:09:19.257325 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.257299    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np"
Jul 17 13:09:19.260568 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.260544    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:09:19.267989 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.267960    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np"]
Jul 17 13:09:19.385106 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.385075    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f573d1b-92b2-47ac-9984-e72625d7f0c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np\" (UID: \"3f573d1b-92b2-47ac-9984-e72625d7f0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np"
Jul 17 13:09:19.486328 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.486297    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f573d1b-92b2-47ac-9984-e72625d7f0c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np\" (UID: \"3f573d1b-92b2-47ac-9984-e72625d7f0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np"
Jul 17 13:09:19.486933 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.486915    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f573d1b-92b2-47ac-9984-e72625d7f0c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np\" (UID: \"3f573d1b-92b2-47ac-9984-e72625d7f0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np"
Jul 17 13:09:19.569842 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.569781    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np"
Jul 17 13:09:19.716920 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:19.716894    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np"]
Jul 17 13:09:19.718560 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:09:19.718535    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f573d1b_92b2_47ac_9984_e72625d7f0c0.slice/crio-670d6f18438e4f1fe117d6a2260b27723c86858c6ec89c00660c625895bbef67 WatchSource:0}: Error finding container 670d6f18438e4f1fe117d6a2260b27723c86858c6ec89c00660c625895bbef67: Status 404 returned error can't find the container with id 670d6f18438e4f1fe117d6a2260b27723c86858c6ec89c00660c625895bbef67
Jul 17 13:09:20.432003 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:20.431979    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5"]
Jul 17 13:09:20.435201 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:20.435183    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5"
Jul 17 13:09:20.445672 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:20.445650    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5"]
Jul 17 13:09:20.494135 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:20.494111    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6851dbaf-33f3-427a-8857-36259fa8e788-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5\" (UID: \"6851dbaf-33f3-427a-8857-36259fa8e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5"
Jul 17 13:09:20.594985 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:20.594934    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6851dbaf-33f3-427a-8857-36259fa8e788-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5\" (UID: \"6851dbaf-33f3-427a-8857-36259fa8e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5"
Jul 17 13:09:20.595503 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:20.595486    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6851dbaf-33f3-427a-8857-36259fa8e788-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5\" (UID: \"6851dbaf-33f3-427a-8857-36259fa8e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5"
Jul 17 13:09:20.681815 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:20.681791    2569 generic.go:356] "Generic (PLEG): container finished" podID="3f573d1b-92b2-47ac-9984-e72625d7f0c0" containerID="e45e2f4e3a1a4e355c6b922a90db8fa0266c03fd0ecb75974678a52ab3a69b5d" exitCode=7
Jul 17 13:09:20.681896 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:20.681818    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np" event={"ID":"3f573d1b-92b2-47ac-9984-e72625d7f0c0","Type":"ContainerDied","Data":"e45e2f4e3a1a4e355c6b922a90db8fa0266c03fd0ecb75974678a52ab3a69b5d"}
Jul 17 13:09:20.681896 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:20.681863    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np" event={"ID":"3f573d1b-92b2-47ac-9984-e72625d7f0c0","Type":"ContainerStarted","Data":"670d6f18438e4f1fe117d6a2260b27723c86858c6ec89c00660c625895bbef67"}
Jul 17 13:09:20.682019 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:20.682007    2569 scope.go:117] "RemoveContainer" containerID="e45e2f4e3a1a4e355c6b922a90db8fa0266c03fd0ecb75974678a52ab3a69b5d"
Jul 17 13:09:20.746122 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:20.746103    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5"
Jul 17 13:09:20.875165 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:20.875102    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5"]
Jul 17 13:09:20.878452 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:09:20.878426    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6851dbaf_33f3_427a_8857_36259fa8e788.slice/crio-5ccf0f3cb6882e8ee6eb96e3ddef51b0bf91ef0caf8e758fbda17bac4b3c0b76 WatchSource:0}: Error finding container 5ccf0f3cb6882e8ee6eb96e3ddef51b0bf91ef0caf8e758fbda17bac4b3c0b76: Status 404 returned error can't find the container with id 5ccf0f3cb6882e8ee6eb96e3ddef51b0bf91ef0caf8e758fbda17bac4b3c0b76
Jul 17 13:09:21.687125 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:21.687090    2569 generic.go:356] "Generic (PLEG): container finished" podID="6851dbaf-33f3-427a-8857-36259fa8e788" containerID="4ecfd59383bed076ee94be893e4e1a89c3aac68dc09fe4b8b5d82b927da96f65" exitCode=6
Jul 17 13:09:21.687434 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:21.687180    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5" event={"ID":"6851dbaf-33f3-427a-8857-36259fa8e788","Type":"ContainerDied","Data":"4ecfd59383bed076ee94be893e4e1a89c3aac68dc09fe4b8b5d82b927da96f65"}
Jul 17 13:09:21.687434 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:21.687235    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5" event={"ID":"6851dbaf-33f3-427a-8857-36259fa8e788","Type":"ContainerStarted","Data":"5ccf0f3cb6882e8ee6eb96e3ddef51b0bf91ef0caf8e758fbda17bac4b3c0b76"}
Jul 17 13:09:21.687434 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:21.687331    2569 scope.go:117] "RemoveContainer" containerID="4ecfd59383bed076ee94be893e4e1a89c3aac68dc09fe4b8b5d82b927da96f65"
Jul 17 13:09:21.689052 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:21.689028    2569 generic.go:356] "Generic (PLEG): container finished" podID="3f573d1b-92b2-47ac-9984-e72625d7f0c0" containerID="874dd8d3710cfbe0c42c2fdc638cbe653e2bea49f2c794a450c112acbe02686c" exitCode=7
Jul 17 13:09:21.689156 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:21.689062    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np" event={"ID":"3f573d1b-92b2-47ac-9984-e72625d7f0c0","Type":"ContainerDied","Data":"874dd8d3710cfbe0c42c2fdc638cbe653e2bea49f2c794a450c112acbe02686c"}
Jul 17 13:09:21.689156 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:21.689101    2569 scope.go:117] "RemoveContainer" containerID="e45e2f4e3a1a4e355c6b922a90db8fa0266c03fd0ecb75974678a52ab3a69b5d"
Jul 17 13:09:21.689315 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:21.689291    2569 scope.go:117] "RemoveContainer" containerID="874dd8d3710cfbe0c42c2fdc638cbe653e2bea49f2c794a450c112acbe02686c"
Jul 17 13:09:21.689510 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:09:21.689477    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np_odh-ai-gateway-infra(3f573d1b-92b2-47ac-9984-e72625d7f0c0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np" podUID="3f573d1b-92b2-47ac-9984-e72625d7f0c0"
Jul 17 13:09:22.694875 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:22.694805    2569 generic.go:356] "Generic (PLEG): container finished" podID="6851dbaf-33f3-427a-8857-36259fa8e788" containerID="917698dc767bc82f40b6d0e07014e4acc84536560b6f92b57fa93b337bc5e31c" exitCode=6
Jul 17 13:09:22.695313 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:22.694898    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5" event={"ID":"6851dbaf-33f3-427a-8857-36259fa8e788","Type":"ContainerDied","Data":"917698dc767bc82f40b6d0e07014e4acc84536560b6f92b57fa93b337bc5e31c"}
Jul 17 13:09:22.695313 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:22.694954    2569 scope.go:117] "RemoveContainer" containerID="4ecfd59383bed076ee94be893e4e1a89c3aac68dc09fe4b8b5d82b927da96f65"
Jul 17 13:09:22.695313 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:22.695047    2569 scope.go:117] "RemoveContainer" containerID="917698dc767bc82f40b6d0e07014e4acc84536560b6f92b57fa93b337bc5e31c"
Jul 17 13:09:22.695313 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:09:22.695225    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5_odh-ai-gateway-infra(6851dbaf-33f3-427a-8857-36259fa8e788)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5" podUID="6851dbaf-33f3-427a-8857-36259fa8e788"
Jul 17 13:09:22.696589 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:22.696572    2569 scope.go:117] "RemoveContainer" containerID="874dd8d3710cfbe0c42c2fdc638cbe653e2bea49f2c794a450c112acbe02686c"
Jul 17 13:09:22.696765 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:09:22.696734    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np_odh-ai-gateway-infra(3f573d1b-92b2-47ac-9984-e72625d7f0c0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np" podUID="3f573d1b-92b2-47ac-9984-e72625d7f0c0"
Jul 17 13:09:23.628791 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:23.628737    2569 scope.go:117] "RemoveContainer" containerID="3d3bd53ccc87d38e5ac00218c2e91639dd179589f67a763b0ddca1d8324d4ae1"
Jul 17 13:09:23.628969 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:09:23.628941    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:09:23.702722 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:23.702698    2569 scope.go:117] "RemoveContainer" containerID="917698dc767bc82f40b6d0e07014e4acc84536560b6f92b57fa93b337bc5e31c"
Jul 17 13:09:23.703098 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:09:23.702863    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5_odh-ai-gateway-infra(6851dbaf-33f3-427a-8857-36259fa8e788)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5" podUID="6851dbaf-33f3-427a-8857-36259fa8e788"
Jul 17 13:09:34.628982 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:34.628948    2569 scope.go:117] "RemoveContainer" containerID="874dd8d3710cfbe0c42c2fdc638cbe653e2bea49f2c794a450c112acbe02686c"
Jul 17 13:09:35.630331 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:35.630304    2569 scope.go:117] "RemoveContainer" containerID="917698dc767bc82f40b6d0e07014e4acc84536560b6f92b57fa93b337bc5e31c"
Jul 17 13:09:35.754670 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:35.754646    2569 generic.go:356] "Generic (PLEG): container finished" podID="3f573d1b-92b2-47ac-9984-e72625d7f0c0" containerID="ea248a8b9219291cd14df7f45d54fdd7ab37d5ffaaa2e3f6699124bbfb0a26a7" exitCode=7
Jul 17 13:09:35.754786 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:35.754715    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np" event={"ID":"3f573d1b-92b2-47ac-9984-e72625d7f0c0","Type":"ContainerDied","Data":"ea248a8b9219291cd14df7f45d54fdd7ab37d5ffaaa2e3f6699124bbfb0a26a7"}
Jul 17 13:09:35.754836 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:35.754793    2569 scope.go:117] "RemoveContainer" containerID="874dd8d3710cfbe0c42c2fdc638cbe653e2bea49f2c794a450c112acbe02686c"
Jul 17 13:09:35.755138 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:35.755117    2569 scope.go:117] "RemoveContainer" containerID="ea248a8b9219291cd14df7f45d54fdd7ab37d5ffaaa2e3f6699124bbfb0a26a7"
Jul 17 13:09:35.755296 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:09:35.755275    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np_odh-ai-gateway-infra(3f573d1b-92b2-47ac-9984-e72625d7f0c0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np" podUID="3f573d1b-92b2-47ac-9984-e72625d7f0c0"
Jul 17 13:09:36.761430 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:36.761395    2569 generic.go:356] "Generic (PLEG): container finished" podID="6851dbaf-33f3-427a-8857-36259fa8e788" containerID="8d1072b9efb8159f3a60fb343ed0a2daa4f187c8c5e429e059acebe031d01da7" exitCode=6
Jul 17 13:09:36.761932 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:36.761465    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5" event={"ID":"6851dbaf-33f3-427a-8857-36259fa8e788","Type":"ContainerDied","Data":"8d1072b9efb8159f3a60fb343ed0a2daa4f187c8c5e429e059acebe031d01da7"}
Jul 17 13:09:36.761932 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:36.761523    2569 scope.go:117] "RemoveContainer" containerID="917698dc767bc82f40b6d0e07014e4acc84536560b6f92b57fa93b337bc5e31c"
Jul 17 13:09:36.761932 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:36.761837    2569 scope.go:117] "RemoveContainer" containerID="8d1072b9efb8159f3a60fb343ed0a2daa4f187c8c5e429e059acebe031d01da7"
Jul 17 13:09:36.762121 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:09:36.762041    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5_odh-ai-gateway-infra(6851dbaf-33f3-427a-8857-36259fa8e788)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5" podUID="6851dbaf-33f3-427a-8857-36259fa8e788"
Jul 17 13:09:36.789513 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:36.789491    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np"]
Jul 17 13:09:36.924130 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:36.924108    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np"
Jul 17 13:09:37.020718 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:37.020639    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f573d1b-92b2-47ac-9984-e72625d7f0c0-maas-api-service-ca\") pod \"3f573d1b-92b2-47ac-9984-e72625d7f0c0\" (UID: \"3f573d1b-92b2-47ac-9984-e72625d7f0c0\") "
Jul 17 13:09:37.020989 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:37.020968    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f573d1b-92b2-47ac-9984-e72625d7f0c0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f573d1b-92b2-47ac-9984-e72625d7f0c0" (UID: "3f573d1b-92b2-47ac-9984-e72625d7f0c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:09:37.122066 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:37.122041    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f573d1b-92b2-47ac-9984-e72625d7f0c0-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:09:37.770932 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:37.770907    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np" event={"ID":"3f573d1b-92b2-47ac-9984-e72625d7f0c0","Type":"ContainerDied","Data":"670d6f18438e4f1fe117d6a2260b27723c86858c6ec89c00660c625895bbef67"}
Jul 17 13:09:37.770932 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:37.770924    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np"
Jul 17 13:09:37.771361 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:37.770957    2569 scope.go:117] "RemoveContainer" containerID="ea248a8b9219291cd14df7f45d54fdd7ab37d5ffaaa2e3f6699124bbfb0a26a7"
Jul 17 13:09:37.789224 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:37.789202    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np"]
Jul 17 13:09:37.801031 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:37.798939    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5"]
Jul 17 13:09:37.801031 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:37.799044    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-wn8np"]
Jul 17 13:09:37.928315 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:37.928288    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5"
Jul 17 13:09:38.029973 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:38.029946    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6851dbaf-33f3-427a-8857-36259fa8e788-maas-api-service-ca\") pod \"6851dbaf-33f3-427a-8857-36259fa8e788\" (UID: \"6851dbaf-33f3-427a-8857-36259fa8e788\") "
Jul 17 13:09:38.030284 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:38.030264    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6851dbaf-33f3-427a-8857-36259fa8e788-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6851dbaf-33f3-427a-8857-36259fa8e788" (UID: "6851dbaf-33f3-427a-8857-36259fa8e788"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:09:38.130422 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:38.130396    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6851dbaf-33f3-427a-8857-36259fa8e788-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:09:38.628409 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:38.628381    2569 scope.go:117] "RemoveContainer" containerID="3d3bd53ccc87d38e5ac00218c2e91639dd179589f67a763b0ddca1d8324d4ae1"
Jul 17 13:09:38.628643 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:09:38.628615    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:09:38.776942 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:38.776913    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5" event={"ID":"6851dbaf-33f3-427a-8857-36259fa8e788","Type":"ContainerDied","Data":"5ccf0f3cb6882e8ee6eb96e3ddef51b0bf91ef0caf8e758fbda17bac4b3c0b76"}
Jul 17 13:09:38.777316 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:38.776921    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5"
Jul 17 13:09:38.777316 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:38.776960    2569 scope.go:117] "RemoveContainer" containerID="8d1072b9efb8159f3a60fb343ed0a2daa4f187c8c5e429e059acebe031d01da7"
Jul 17 13:09:38.800474 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:38.800451    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5"]
Jul 17 13:09:38.803496 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:38.803476    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xbjb5"]
Jul 17 13:09:39.633991 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:39.633964    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f573d1b-92b2-47ac-9984-e72625d7f0c0" path="/var/lib/kubelet/pods/3f573d1b-92b2-47ac-9984-e72625d7f0c0/volumes"
Jul 17 13:09:39.634312 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:39.634297    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6851dbaf-33f3-427a-8857-36259fa8e788" path="/var/lib/kubelet/pods/6851dbaf-33f3-427a-8857-36259fa8e788/volumes"
Jul 17 13:09:49.629078 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:09:49.629052    2569 scope.go:117] "RemoveContainer" containerID="3d3bd53ccc87d38e5ac00218c2e91639dd179589f67a763b0ddca1d8324d4ae1"
Jul 17 13:09:49.629471 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:09:49.629207    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:10:01.659064 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.658989    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc"]
Jul 17 13:10:01.659498 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.659336    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6851dbaf-33f3-427a-8857-36259fa8e788" containerName="revoke-keys"
Jul 17 13:10:01.659498 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.659349    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6851dbaf-33f3-427a-8857-36259fa8e788" containerName="revoke-keys"
Jul 17 13:10:01.659498 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.659366    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f573d1b-92b2-47ac-9984-e72625d7f0c0" containerName="revoke-keys"
Jul 17 13:10:01.659498 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.659375    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f573d1b-92b2-47ac-9984-e72625d7f0c0" containerName="revoke-keys"
Jul 17 13:10:01.659498 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.659386    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f573d1b-92b2-47ac-9984-e72625d7f0c0" containerName="revoke-keys"
Jul 17 13:10:01.659498 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.659392    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f573d1b-92b2-47ac-9984-e72625d7f0c0" containerName="revoke-keys"
Jul 17 13:10:01.659498 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.659398    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f573d1b-92b2-47ac-9984-e72625d7f0c0" containerName="revoke-keys"
Jul 17 13:10:01.659498 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.659403    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f573d1b-92b2-47ac-9984-e72625d7f0c0" containerName="revoke-keys"
Jul 17 13:10:01.659498 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.659408    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6851dbaf-33f3-427a-8857-36259fa8e788" containerName="revoke-keys"
Jul 17 13:10:01.659498 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.659413    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6851dbaf-33f3-427a-8857-36259fa8e788" containerName="revoke-keys"
Jul 17 13:10:01.659498 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.659421    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6851dbaf-33f3-427a-8857-36259fa8e788" containerName="revoke-keys"
Jul 17 13:10:01.659498 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.659426    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6851dbaf-33f3-427a-8857-36259fa8e788" containerName="revoke-keys"
Jul 17 13:10:01.659498 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.659500    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f573d1b-92b2-47ac-9984-e72625d7f0c0" containerName="revoke-keys"
Jul 17 13:10:01.659928 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.659508    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6851dbaf-33f3-427a-8857-36259fa8e788" containerName="revoke-keys"
Jul 17 13:10:01.659928 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.659515    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f573d1b-92b2-47ac-9984-e72625d7f0c0" containerName="revoke-keys"
Jul 17 13:10:01.659928 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.659520    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6851dbaf-33f3-427a-8857-36259fa8e788" containerName="revoke-keys"
Jul 17 13:10:01.659928 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.659527    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6851dbaf-33f3-427a-8857-36259fa8e788" containerName="revoke-keys"
Jul 17 13:10:01.662515 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.662494    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc"
Jul 17 13:10:01.665286 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.665265    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:10:01.670199 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.670165    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc"]
Jul 17 13:10:01.705999 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.705972    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33298175-d2fa-4c3f-997b-faea87840907-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc\" (UID: \"33298175-d2fa-4c3f-997b-faea87840907\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc"
Jul 17 13:10:01.807409 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.807384    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33298175-d2fa-4c3f-997b-faea87840907-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc\" (UID: \"33298175-d2fa-4c3f-997b-faea87840907\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc"
Jul 17 13:10:01.807969 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.807953    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33298175-d2fa-4c3f-997b-faea87840907-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc\" (UID: \"33298175-d2fa-4c3f-997b-faea87840907\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc"
Jul 17 13:10:01.973308 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:01.973223    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc"
Jul 17 13:10:02.101098 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:02.101075    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc"]
Jul 17 13:10:02.103222 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:10:02.103193    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33298175_d2fa_4c3f_997b_faea87840907.slice/crio-462aae41d347e4ddc308d8506ab426d4565f6f9f2f1b4c88dfc7eb2b9e8e5405 WatchSource:0}: Error finding container 462aae41d347e4ddc308d8506ab426d4565f6f9f2f1b4c88dfc7eb2b9e8e5405: Status 404 returned error can't find the container with id 462aae41d347e4ddc308d8506ab426d4565f6f9f2f1b4c88dfc7eb2b9e8e5405
Jul 17 13:10:02.888958 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:02.888925    2569 generic.go:356] "Generic (PLEG): container finished" podID="33298175-d2fa-4c3f-997b-faea87840907" containerID="7dc0a7a93ca16a96a42037eb7197ee96a038c23094bb9a4559dc1cf637865629" exitCode=6
Jul 17 13:10:02.889310 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:02.888977    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc" event={"ID":"33298175-d2fa-4c3f-997b-faea87840907","Type":"ContainerDied","Data":"7dc0a7a93ca16a96a42037eb7197ee96a038c23094bb9a4559dc1cf637865629"}
Jul 17 13:10:02.889310 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:02.889058    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc" event={"ID":"33298175-d2fa-4c3f-997b-faea87840907","Type":"ContainerStarted","Data":"462aae41d347e4ddc308d8506ab426d4565f6f9f2f1b4c88dfc7eb2b9e8e5405"}
Jul 17 13:10:02.889310 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:02.889219    2569 scope.go:117] "RemoveContainer" containerID="7dc0a7a93ca16a96a42037eb7197ee96a038c23094bb9a4559dc1cf637865629"
Jul 17 13:10:03.895620 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:03.895591    2569 generic.go:356] "Generic (PLEG): container finished" podID="33298175-d2fa-4c3f-997b-faea87840907" containerID="7fd868e8ac683e98b502b857d1584b292018dd8cbe3a95aa367cf5e077d90149" exitCode=6
Jul 17 13:10:03.896090 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:03.895655    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc" event={"ID":"33298175-d2fa-4c3f-997b-faea87840907","Type":"ContainerDied","Data":"7fd868e8ac683e98b502b857d1584b292018dd8cbe3a95aa367cf5e077d90149"}
Jul 17 13:10:03.896090 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:03.895732    2569 scope.go:117] "RemoveContainer" containerID="7dc0a7a93ca16a96a42037eb7197ee96a038c23094bb9a4559dc1cf637865629"
Jul 17 13:10:03.896090 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:03.895916    2569 scope.go:117] "RemoveContainer" containerID="7fd868e8ac683e98b502b857d1584b292018dd8cbe3a95aa367cf5e077d90149"
Jul 17 13:10:03.896207 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:10:03.896118    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc_odh-ai-gateway-infra(33298175-d2fa-4c3f-997b-faea87840907)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc" podUID="33298175-d2fa-4c3f-997b-faea87840907"
Jul 17 13:10:04.628819 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:04.628795    2569 scope.go:117] "RemoveContainer" containerID="3d3bd53ccc87d38e5ac00218c2e91639dd179589f67a763b0ddca1d8324d4ae1"
Jul 17 13:10:04.628997 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:10:04.628958    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:10:04.901807 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:04.901702    2569 scope.go:117] "RemoveContainer" containerID="7fd868e8ac683e98b502b857d1584b292018dd8cbe3a95aa367cf5e077d90149"
Jul 17 13:10:04.902172 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:10:04.901905    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc_odh-ai-gateway-infra(33298175-d2fa-4c3f-997b-faea87840907)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc" podUID="33298175-d2fa-4c3f-997b-faea87840907"
Jul 17 13:10:10.474935 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:10.474900    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84"]
Jul 17 13:10:10.475370 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:10.475343    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f573d1b-92b2-47ac-9984-e72625d7f0c0" containerName="revoke-keys"
Jul 17 13:10:10.479637 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:10.479618    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84"
Jul 17 13:10:10.488470 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:10.488442    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84"]
Jul 17 13:10:10.574920 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:10.574893    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248a8332-236f-43a8-a592-e96c79b935df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84\" (UID: \"248a8332-236f-43a8-a592-e96c79b935df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84"
Jul 17 13:10:10.675810 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:10.675782    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248a8332-236f-43a8-a592-e96c79b935df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84\" (UID: \"248a8332-236f-43a8-a592-e96c79b935df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84"
Jul 17 13:10:10.676370 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:10.676348    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248a8332-236f-43a8-a592-e96c79b935df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84\" (UID: \"248a8332-236f-43a8-a592-e96c79b935df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84"
Jul 17 13:10:10.791861 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:10.791804    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84"
Jul 17 13:10:10.922009 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:10.921985    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84"]
Jul 17 13:10:10.924111 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:10:10.924069    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248a8332_236f_43a8_a592_e96c79b935df.slice/crio-bb808e8bf10033bfa4fcc3251c46e213a140363ec4336464accd908a2b771360 WatchSource:0}: Error finding container bb808e8bf10033bfa4fcc3251c46e213a140363ec4336464accd908a2b771360: Status 404 returned error can't find the container with id bb808e8bf10033bfa4fcc3251c46e213a140363ec4336464accd908a2b771360
Jul 17 13:10:10.934525 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:10.934502    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84" event={"ID":"248a8332-236f-43a8-a592-e96c79b935df","Type":"ContainerStarted","Data":"bb808e8bf10033bfa4fcc3251c46e213a140363ec4336464accd908a2b771360"}
Jul 17 13:10:11.940478 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:11.940444    2569 generic.go:356] "Generic (PLEG): container finished" podID="248a8332-236f-43a8-a592-e96c79b935df" containerID="1c4c1e0320ebc03afba28c8818c0508265016c7fd6d75af58cb61dd85b35d71a" exitCode=6
Jul 17 13:10:11.940897 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:11.940538    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84" event={"ID":"248a8332-236f-43a8-a592-e96c79b935df","Type":"ContainerDied","Data":"1c4c1e0320ebc03afba28c8818c0508265016c7fd6d75af58cb61dd85b35d71a"}
Jul 17 13:10:11.940897 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:11.940677    2569 scope.go:117] "RemoveContainer" containerID="1c4c1e0320ebc03afba28c8818c0508265016c7fd6d75af58cb61dd85b35d71a"
Jul 17 13:10:12.946623 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:12.946554    2569 generic.go:356] "Generic (PLEG): container finished" podID="248a8332-236f-43a8-a592-e96c79b935df" containerID="4c21a6f5311e967e0c0352bf8f554c03dd46b8db3306226efcb87b67793752ce" exitCode=6
Jul 17 13:10:12.946960 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:12.946653    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84" event={"ID":"248a8332-236f-43a8-a592-e96c79b935df","Type":"ContainerDied","Data":"4c21a6f5311e967e0c0352bf8f554c03dd46b8db3306226efcb87b67793752ce"}
Jul 17 13:10:12.946960 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:12.946717    2569 scope.go:117] "RemoveContainer" containerID="1c4c1e0320ebc03afba28c8818c0508265016c7fd6d75af58cb61dd85b35d71a"
Jul 17 13:10:12.946960 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:12.946809    2569 scope.go:117] "RemoveContainer" containerID="4c21a6f5311e967e0c0352bf8f554c03dd46b8db3306226efcb87b67793752ce"
Jul 17 13:10:12.947073 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:10:12.946976    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84_odh-ai-gateway-infra(248a8332-236f-43a8-a592-e96c79b935df)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84" podUID="248a8332-236f-43a8-a592-e96c79b935df"
Jul 17 13:10:13.951941 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:13.951916    2569 scope.go:117] "RemoveContainer" containerID="4c21a6f5311e967e0c0352bf8f554c03dd46b8db3306226efcb87b67793752ce"
Jul 17 13:10:13.952370 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:10:13.952154    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84_odh-ai-gateway-infra(248a8332-236f-43a8-a592-e96c79b935df)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84" podUID="248a8332-236f-43a8-a592-e96c79b935df"
Jul 17 13:10:16.628257 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:16.628220    2569 scope.go:117] "RemoveContainer" containerID="7fd868e8ac683e98b502b857d1584b292018dd8cbe3a95aa367cf5e077d90149"
Jul 17 13:10:16.967648 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:16.967620    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc" event={"ID":"33298175-d2fa-4c3f-997b-faea87840907","Type":"ContainerStarted","Data":"ec133153fd2e806288237aed81e29e515e3698eff9e887c3c4046f34d52bfa19"}
Jul 17 13:10:16.988763 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:16.988714    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc" podStartSLOduration=15.988700882 podStartE2EDuration="15.988700882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:10:16.988140972 +0000 UTC m=+2891.992187853" watchObservedRunningTime="2026-07-17 13:10:16.988700882 +0000 UTC m=+2891.992747763"
Jul 17 13:10:17.974325 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:17.974293    2569 generic.go:356] "Generic (PLEG): container finished" podID="33298175-d2fa-4c3f-997b-faea87840907" containerID="ec133153fd2e806288237aed81e29e515e3698eff9e887c3c4046f34d52bfa19" exitCode=6
Jul 17 13:10:17.974792 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:17.974336    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc" event={"ID":"33298175-d2fa-4c3f-997b-faea87840907","Type":"ContainerDied","Data":"ec133153fd2e806288237aed81e29e515e3698eff9e887c3c4046f34d52bfa19"}
Jul 17 13:10:17.974792 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:17.974388    2569 scope.go:117] "RemoveContainer" containerID="7fd868e8ac683e98b502b857d1584b292018dd8cbe3a95aa367cf5e077d90149"
Jul 17 13:10:17.974792 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:17.974692    2569 scope.go:117] "RemoveContainer" containerID="ec133153fd2e806288237aed81e29e515e3698eff9e887c3c4046f34d52bfa19"
Jul 17 13:10:17.974977 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:10:17.974947    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc_odh-ai-gateway-infra(33298175-d2fa-4c3f-997b-faea87840907)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc" podUID="33298175-d2fa-4c3f-997b-faea87840907"
Jul 17 13:10:18.011126 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:18.011098    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc"]
Jul 17 13:10:19.115139 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:19.115118    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc"
Jul 17 13:10:19.143733 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:19.143709    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33298175-d2fa-4c3f-997b-faea87840907-maas-api-service-ca\") pod \"33298175-d2fa-4c3f-997b-faea87840907\" (UID: \"33298175-d2fa-4c3f-997b-faea87840907\") "
Jul 17 13:10:19.144045 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:19.144024    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33298175-d2fa-4c3f-997b-faea87840907-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33298175-d2fa-4c3f-997b-faea87840907" (UID: "33298175-d2fa-4c3f-997b-faea87840907"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:10:19.244778 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:19.244709    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33298175-d2fa-4c3f-997b-faea87840907-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:10:19.628992 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:19.628936    2569 scope.go:117] "RemoveContainer" containerID="3d3bd53ccc87d38e5ac00218c2e91639dd179589f67a763b0ddca1d8324d4ae1"
Jul 17 13:10:19.987335 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:19.987304    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc" event={"ID":"33298175-d2fa-4c3f-997b-faea87840907","Type":"ContainerDied","Data":"462aae41d347e4ddc308d8506ab426d4565f6f9f2f1b4c88dfc7eb2b9e8e5405"}
Jul 17 13:10:19.987335 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:19.987321    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc"
Jul 17 13:10:19.987533 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:19.987359    2569 scope.go:117] "RemoveContainer" containerID="ec133153fd2e806288237aed81e29e515e3698eff9e887c3c4046f34d52bfa19"
Jul 17 13:10:20.009709 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:20.009688    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc"]
Jul 17 13:10:20.012953 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:20.012932    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qx8qc"]
Jul 17 13:10:20.994377 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:20.994347    2569 generic.go:356] "Generic (PLEG): container finished" podID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerID="401a7ae3df3aa2bf10592879840e334cccd00cdc103da71b62715d1859f9287d" exitCode=1
Jul 17 13:10:20.994830 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:20.994420    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" event={"ID":"bcff1798-e8d8-4449-b8e3-eaf723b636c4","Type":"ContainerDied","Data":"401a7ae3df3aa2bf10592879840e334cccd00cdc103da71b62715d1859f9287d"}
Jul 17 13:10:20.994830 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:20.994487    2569 scope.go:117] "RemoveContainer" containerID="3d3bd53ccc87d38e5ac00218c2e91639dd179589f67a763b0ddca1d8324d4ae1"
Jul 17 13:10:20.994830 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:20.994827    2569 scope.go:117] "RemoveContainer" containerID="401a7ae3df3aa2bf10592879840e334cccd00cdc103da71b62715d1859f9287d"
Jul 17 13:10:20.995059 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:10:20.995036    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:10:21.270961 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:21.270891    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:10:21.634426 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:21.634345    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33298175-d2fa-4c3f-997b-faea87840907" path="/var/lib/kubelet/pods/33298175-d2fa-4c3f-997b-faea87840907/volumes"
Jul 17 13:10:22.002147 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:22.002123    2569 scope.go:117] "RemoveContainer" containerID="401a7ae3df3aa2bf10592879840e334cccd00cdc103da71b62715d1859f9287d"
Jul 17 13:10:22.002537 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:10:22.002287    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:10:25.388956 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:25.388909    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:10:25.389370 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:25.389275    2569 scope.go:117] "RemoveContainer" containerID="401a7ae3df3aa2bf10592879840e334cccd00cdc103da71b62715d1859f9287d"
Jul 17 13:10:25.389450 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:10:25.389432    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:10:25.631869 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:25.631844    2569 scope.go:117] "RemoveContainer" containerID="4c21a6f5311e967e0c0352bf8f554c03dd46b8db3306226efcb87b67793752ce"
Jul 17 13:10:26.023629 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:26.023602    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84" event={"ID":"248a8332-236f-43a8-a592-e96c79b935df","Type":"ContainerStarted","Data":"c64e4f7eab87af7ed731f8af461930bbe503558cde7c80d519ce61a6d79fc350"}
Jul 17 13:10:26.042580 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:26.042540    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84" podStartSLOduration=16.04252558 podStartE2EDuration="16.04252558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:10:26.041000487 +0000 UTC m=+2901.045047372" watchObservedRunningTime="2026-07-17 13:10:26.04252558 +0000 UTC m=+2901.046572461"
Jul 17 13:10:27.030071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:27.030039    2569 generic.go:356] "Generic (PLEG): container finished" podID="248a8332-236f-43a8-a592-e96c79b935df" containerID="c64e4f7eab87af7ed731f8af461930bbe503558cde7c80d519ce61a6d79fc350" exitCode=6
Jul 17 13:10:27.030478 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:27.030119    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84" event={"ID":"248a8332-236f-43a8-a592-e96c79b935df","Type":"ContainerDied","Data":"c64e4f7eab87af7ed731f8af461930bbe503558cde7c80d519ce61a6d79fc350"}
Jul 17 13:10:27.030478 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:27.030185    2569 scope.go:117] "RemoveContainer" containerID="4c21a6f5311e967e0c0352bf8f554c03dd46b8db3306226efcb87b67793752ce"
Jul 17 13:10:27.030478 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:27.030425    2569 scope.go:117] "RemoveContainer" containerID="c64e4f7eab87af7ed731f8af461930bbe503558cde7c80d519ce61a6d79fc350"
Jul 17 13:10:27.030666 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:10:27.030641    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84_odh-ai-gateway-infra(248a8332-236f-43a8-a592-e96c79b935df)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84" podUID="248a8332-236f-43a8-a592-e96c79b935df"
Jul 17 13:10:27.058012 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:27.057988    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84"]
Jul 17 13:10:28.169086 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:28.169063    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84"
Jul 17 13:10:28.211990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:28.211969    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248a8332-236f-43a8-a592-e96c79b935df-maas-api-service-ca\") pod \"248a8332-236f-43a8-a592-e96c79b935df\" (UID: \"248a8332-236f-43a8-a592-e96c79b935df\") "
Jul 17 13:10:28.212287 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:28.212265    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248a8332-236f-43a8-a592-e96c79b935df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "248a8332-236f-43a8-a592-e96c79b935df" (UID: "248a8332-236f-43a8-a592-e96c79b935df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:10:28.313497 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:28.313440    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248a8332-236f-43a8-a592-e96c79b935df-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:10:29.042186 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:29.042144    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84" event={"ID":"248a8332-236f-43a8-a592-e96c79b935df","Type":"ContainerDied","Data":"bb808e8bf10033bfa4fcc3251c46e213a140363ec4336464accd908a2b771360"}
Jul 17 13:10:29.042186 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:29.042169    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84"
Jul 17 13:10:29.042431 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:29.042218    2569 scope.go:117] "RemoveContainer" containerID="c64e4f7eab87af7ed731f8af461930bbe503558cde7c80d519ce61a6d79fc350"
Jul 17 13:10:29.064613 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:29.064589    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84"]
Jul 17 13:10:29.068778 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:29.068741    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-lkf84"]
Jul 17 13:10:29.640195 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:29.640152    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="248a8332-236f-43a8-a592-e96c79b935df" path="/var/lib/kubelet/pods/248a8332-236f-43a8-a592-e96c79b935df/volumes"
Jul 17 13:10:37.629383 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:37.629335    2569 scope.go:117] "RemoveContainer" containerID="401a7ae3df3aa2bf10592879840e334cccd00cdc103da71b62715d1859f9287d"
Jul 17 13:10:37.629959 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:10:37.629539    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:10:48.628723 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:48.628689    2569 scope.go:117] "RemoveContainer" containerID="401a7ae3df3aa2bf10592879840e334cccd00cdc103da71b62715d1859f9287d"
Jul 17 13:10:48.629187 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:10:48.628906    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:10:51.706542 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.706509    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6"]
Jul 17 13:10:51.706943 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.706830    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="248a8332-236f-43a8-a592-e96c79b935df" containerName="revoke-keys"
Jul 17 13:10:51.706943 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.706842    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="248a8332-236f-43a8-a592-e96c79b935df" containerName="revoke-keys"
Jul 17 13:10:51.706943 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.706855    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="248a8332-236f-43a8-a592-e96c79b935df" containerName="revoke-keys"
Jul 17 13:10:51.706943 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.706861    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="248a8332-236f-43a8-a592-e96c79b935df" containerName="revoke-keys"
Jul 17 13:10:51.706943 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.706871    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33298175-d2fa-4c3f-997b-faea87840907" containerName="revoke-keys"
Jul 17 13:10:51.706943 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.706876    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="33298175-d2fa-4c3f-997b-faea87840907" containerName="revoke-keys"
Jul 17 13:10:51.706943 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.706884    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33298175-d2fa-4c3f-997b-faea87840907" containerName="revoke-keys"
Jul 17 13:10:51.706943 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.706888    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="33298175-d2fa-4c3f-997b-faea87840907" containerName="revoke-keys"
Jul 17 13:10:51.706943 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.706896    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33298175-d2fa-4c3f-997b-faea87840907" containerName="revoke-keys"
Jul 17 13:10:51.706943 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.706902    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="33298175-d2fa-4c3f-997b-faea87840907" containerName="revoke-keys"
Jul 17 13:10:51.706943 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.706907    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="248a8332-236f-43a8-a592-e96c79b935df" containerName="revoke-keys"
Jul 17 13:10:51.706943 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.706913    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="248a8332-236f-43a8-a592-e96c79b935df" containerName="revoke-keys"
Jul 17 13:10:51.707317 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.706971    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="248a8332-236f-43a8-a592-e96c79b935df" containerName="revoke-keys"
Jul 17 13:10:51.707317 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.706982    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="33298175-d2fa-4c3f-997b-faea87840907" containerName="revoke-keys"
Jul 17 13:10:51.707317 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.706990    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="33298175-d2fa-4c3f-997b-faea87840907" containerName="revoke-keys"
Jul 17 13:10:51.707317 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.707001    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="248a8332-236f-43a8-a592-e96c79b935df" containerName="revoke-keys"
Jul 17 13:10:51.707317 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.707010    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="33298175-d2fa-4c3f-997b-faea87840907" containerName="revoke-keys"
Jul 17 13:10:51.710016 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.709988    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6"
Jul 17 13:10:51.712801 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.712779    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:10:51.717844 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.717816    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6"]
Jul 17 13:10:51.788637 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.788617    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51796b11-3bdc-4aff-bca5-02182bccae85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6\" (UID: \"51796b11-3bdc-4aff-bca5-02182bccae85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6"
Jul 17 13:10:51.889268 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.889244    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51796b11-3bdc-4aff-bca5-02182bccae85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6\" (UID: \"51796b11-3bdc-4aff-bca5-02182bccae85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6"
Jul 17 13:10:51.889829 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:51.889812    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51796b11-3bdc-4aff-bca5-02182bccae85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6\" (UID: \"51796b11-3bdc-4aff-bca5-02182bccae85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6"
Jul 17 13:10:52.021876 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:52.021852    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6"
Jul 17 13:10:52.152342 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:52.152318    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6"]
Jul 17 13:10:52.154360 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:10:52.154331    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51796b11_3bdc_4aff_bca5_02182bccae85.slice/crio-24c3f3cec6ef7fa03564117134135d245e66a03405177b0297399b7b5e9b498f WatchSource:0}: Error finding container 24c3f3cec6ef7fa03564117134135d245e66a03405177b0297399b7b5e9b498f: Status 404 returned error can't find the container with id 24c3f3cec6ef7fa03564117134135d245e66a03405177b0297399b7b5e9b498f
Jul 17 13:10:53.152680 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:53.152617    2569 generic.go:356] "Generic (PLEG): container finished" podID="51796b11-3bdc-4aff-bca5-02182bccae85" containerID="9713be50f10d747bfb8e3788d9304e25c812f26d1a061f42364d6a7e17fbc3e3" exitCode=6
Jul 17 13:10:53.153047 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:53.152705    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6" event={"ID":"51796b11-3bdc-4aff-bca5-02182bccae85","Type":"ContainerDied","Data":"9713be50f10d747bfb8e3788d9304e25c812f26d1a061f42364d6a7e17fbc3e3"}
Jul 17 13:10:53.153047 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:53.152791    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6" event={"ID":"51796b11-3bdc-4aff-bca5-02182bccae85","Type":"ContainerStarted","Data":"24c3f3cec6ef7fa03564117134135d245e66a03405177b0297399b7b5e9b498f"}
Jul 17 13:10:53.153047 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:53.152842    2569 scope.go:117] "RemoveContainer" containerID="9713be50f10d747bfb8e3788d9304e25c812f26d1a061f42364d6a7e17fbc3e3"
Jul 17 13:10:54.159861 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:54.159827    2569 generic.go:356] "Generic (PLEG): container finished" podID="51796b11-3bdc-4aff-bca5-02182bccae85" containerID="04f7969aab7532a08d76efacd56f2f10836eb2329d6b1fba52bb08efaa488906" exitCode=6
Jul 17 13:10:54.160226 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:54.159910    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6" event={"ID":"51796b11-3bdc-4aff-bca5-02182bccae85","Type":"ContainerDied","Data":"04f7969aab7532a08d76efacd56f2f10836eb2329d6b1fba52bb08efaa488906"}
Jul 17 13:10:54.160226 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:54.159974    2569 scope.go:117] "RemoveContainer" containerID="9713be50f10d747bfb8e3788d9304e25c812f26d1a061f42364d6a7e17fbc3e3"
Jul 17 13:10:54.160226 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:54.160130    2569 scope.go:117] "RemoveContainer" containerID="04f7969aab7532a08d76efacd56f2f10836eb2329d6b1fba52bb08efaa488906"
Jul 17 13:10:54.160409 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:10:54.160322    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6_odh-ai-gateway-infra(51796b11-3bdc-4aff-bca5-02182bccae85)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6" podUID="51796b11-3bdc-4aff-bca5-02182bccae85"
Jul 17 13:10:55.167381 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:10:55.167357    2569 scope.go:117] "RemoveContainer" containerID="04f7969aab7532a08d76efacd56f2f10836eb2329d6b1fba52bb08efaa488906"
Jul 17 13:10:55.167792 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:10:55.167524    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6_odh-ai-gateway-infra(51796b11-3bdc-4aff-bca5-02182bccae85)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6" podUID="51796b11-3bdc-4aff-bca5-02182bccae85"
Jul 17 13:11:00.528444 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:00.528402    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8"]
Jul 17 13:11:00.528908 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:00.528860    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="248a8332-236f-43a8-a592-e96c79b935df" containerName="revoke-keys"
Jul 17 13:11:00.533404 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:00.533381    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8"
Jul 17 13:11:00.548946 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:00.548915    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8"]
Jul 17 13:11:00.628613 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:00.628587    2569 scope.go:117] "RemoveContainer" containerID="401a7ae3df3aa2bf10592879840e334cccd00cdc103da71b62715d1859f9287d"
Jul 17 13:11:00.628807 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:11:00.628780    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:11:00.661625 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:00.661599    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ce8b8b-ce9c-4de4-93bf-dd0c052385c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8\" (UID: \"21ce8b8b-ce9c-4de4-93bf-dd0c052385c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8"
Jul 17 13:11:00.762500 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:00.762477    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ce8b8b-ce9c-4de4-93bf-dd0c052385c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8\" (UID: \"21ce8b8b-ce9c-4de4-93bf-dd0c052385c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8"
Jul 17 13:11:00.763167 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:00.763142    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ce8b8b-ce9c-4de4-93bf-dd0c052385c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8\" (UID: \"21ce8b8b-ce9c-4de4-93bf-dd0c052385c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8"
Jul 17 13:11:00.843766 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:00.843700    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8"
Jul 17 13:11:00.978667 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:00.978636    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8"]
Jul 17 13:11:00.982152 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:11:00.982127    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ce8b8b_ce9c_4de4_93bf_dd0c052385c6.slice/crio-bc76e68661afa8a0b71bf4b7c6538dd5be7cfdbd9ac43934a342b799bfc1391e WatchSource:0}: Error finding container bc76e68661afa8a0b71bf4b7c6538dd5be7cfdbd9ac43934a342b799bfc1391e: Status 404 returned error can't find the container with id bc76e68661afa8a0b71bf4b7c6538dd5be7cfdbd9ac43934a342b799bfc1391e
Jul 17 13:11:01.195930 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:01.195905    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8" event={"ID":"21ce8b8b-ce9c-4de4-93bf-dd0c052385c6","Type":"ContainerStarted","Data":"c2149ea5b30040997ac3c0647c7931687267e805e09c4ca5e0699799aa5e546a"}
Jul 17 13:11:01.196019 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:01.195956    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8" event={"ID":"21ce8b8b-ce9c-4de4-93bf-dd0c052385c6","Type":"ContainerStarted","Data":"bc76e68661afa8a0b71bf4b7c6538dd5be7cfdbd9ac43934a342b799bfc1391e"}
Jul 17 13:11:01.214972 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:01.214930    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8" podStartSLOduration=1.214915436 podStartE2EDuration="1.214915436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:11:01.213864634 +0000 UTC m=+2936.217911517" watchObservedRunningTime="2026-07-17 13:11:01.214915436 +0000 UTC m=+2936.218962378"
Jul 17 13:11:02.202468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:02.202435    2569 generic.go:356] "Generic (PLEG): container finished" podID="21ce8b8b-ce9c-4de4-93bf-dd0c052385c6" containerID="c2149ea5b30040997ac3c0647c7931687267e805e09c4ca5e0699799aa5e546a" exitCode=6
Jul 17 13:11:02.202811 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:02.202490    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8" event={"ID":"21ce8b8b-ce9c-4de4-93bf-dd0c052385c6","Type":"ContainerDied","Data":"c2149ea5b30040997ac3c0647c7931687267e805e09c4ca5e0699799aa5e546a"}
Jul 17 13:11:02.202811 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:02.202694    2569 scope.go:117] "RemoveContainer" containerID="c2149ea5b30040997ac3c0647c7931687267e805e09c4ca5e0699799aa5e546a"
Jul 17 13:11:03.209685 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:03.209610    2569 generic.go:356] "Generic (PLEG): container finished" podID="21ce8b8b-ce9c-4de4-93bf-dd0c052385c6" containerID="1605838e4ede9ca3778943cec2d98b0e7e5e941d9b36252b8210745a8ff7fb5f" exitCode=6
Jul 17 13:11:03.210104 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:03.209697    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8" event={"ID":"21ce8b8b-ce9c-4de4-93bf-dd0c052385c6","Type":"ContainerDied","Data":"1605838e4ede9ca3778943cec2d98b0e7e5e941d9b36252b8210745a8ff7fb5f"}
Jul 17 13:11:03.210104 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:03.209806    2569 scope.go:117] "RemoveContainer" containerID="c2149ea5b30040997ac3c0647c7931687267e805e09c4ca5e0699799aa5e546a"
Jul 17 13:11:03.210104 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:03.209880    2569 scope.go:117] "RemoveContainer" containerID="1605838e4ede9ca3778943cec2d98b0e7e5e941d9b36252b8210745a8ff7fb5f"
Jul 17 13:11:03.210104 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:11:03.210064    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8_odh-ai-gateway-infra(21ce8b8b-ce9c-4de4-93bf-dd0c052385c6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8" podUID="21ce8b8b-ce9c-4de4-93bf-dd0c052385c6"
Jul 17 13:11:04.215864 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:04.215842    2569 scope.go:117] "RemoveContainer" containerID="1605838e4ede9ca3778943cec2d98b0e7e5e941d9b36252b8210745a8ff7fb5f"
Jul 17 13:11:04.216237 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:11:04.215989    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8_odh-ai-gateway-infra(21ce8b8b-ce9c-4de4-93bf-dd0c052385c6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8" podUID="21ce8b8b-ce9c-4de4-93bf-dd0c052385c6"
Jul 17 13:11:08.628819 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:08.628781    2569 scope.go:117] "RemoveContainer" containerID="04f7969aab7532a08d76efacd56f2f10836eb2329d6b1fba52bb08efaa488906"
Jul 17 13:11:09.239953 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:09.239929    2569 generic.go:356] "Generic (PLEG): container finished" podID="51796b11-3bdc-4aff-bca5-02182bccae85" containerID="47b03f0be7734dfc0b484aa64038e55f4ea5b7351d5e7c047306d99e7ba30994" exitCode=6
Jul 17 13:11:09.240068 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:09.239987    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6" event={"ID":"51796b11-3bdc-4aff-bca5-02182bccae85","Type":"ContainerDied","Data":"47b03f0be7734dfc0b484aa64038e55f4ea5b7351d5e7c047306d99e7ba30994"}
Jul 17 13:11:09.240068 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:09.240034    2569 scope.go:117] "RemoveContainer" containerID="04f7969aab7532a08d76efacd56f2f10836eb2329d6b1fba52bb08efaa488906"
Jul 17 13:11:09.240316 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:09.240298    2569 scope.go:117] "RemoveContainer" containerID="47b03f0be7734dfc0b484aa64038e55f4ea5b7351d5e7c047306d99e7ba30994"
Jul 17 13:11:09.240496 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:11:09.240474    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6_odh-ai-gateway-infra(51796b11-3bdc-4aff-bca5-02182bccae85)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6" podUID="51796b11-3bdc-4aff-bca5-02182bccae85"
Jul 17 13:11:10.270059 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:10.270031    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6"]
Jul 17 13:11:10.404648 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:10.404624    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6"
Jul 17 13:11:10.538997 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:10.538924    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51796b11-3bdc-4aff-bca5-02182bccae85-maas-api-service-ca\") pod \"51796b11-3bdc-4aff-bca5-02182bccae85\" (UID: \"51796b11-3bdc-4aff-bca5-02182bccae85\") "
Jul 17 13:11:10.539242 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:10.539220    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51796b11-3bdc-4aff-bca5-02182bccae85-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "51796b11-3bdc-4aff-bca5-02182bccae85" (UID: "51796b11-3bdc-4aff-bca5-02182bccae85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:11:10.640088 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:10.640062    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51796b11-3bdc-4aff-bca5-02182bccae85-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:11:11.252384 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:11.252358    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6"
Jul 17 13:11:11.252548 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:11.252353    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6" event={"ID":"51796b11-3bdc-4aff-bca5-02182bccae85","Type":"ContainerDied","Data":"24c3f3cec6ef7fa03564117134135d245e66a03405177b0297399b7b5e9b498f"}
Jul 17 13:11:11.252548 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:11.252498    2569 scope.go:117] "RemoveContainer" containerID="47b03f0be7734dfc0b484aa64038e55f4ea5b7351d5e7c047306d99e7ba30994"
Jul 17 13:11:11.291496 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:11.291474    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6"]
Jul 17 13:11:11.298707 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:11.298685    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-xbgb6"]
Jul 17 13:11:11.633943 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:11.633879    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51796b11-3bdc-4aff-bca5-02182bccae85" path="/var/lib/kubelet/pods/51796b11-3bdc-4aff-bca5-02182bccae85/volumes"
Jul 17 13:11:14.628771 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:14.628731    2569 scope.go:117] "RemoveContainer" containerID="401a7ae3df3aa2bf10592879840e334cccd00cdc103da71b62715d1859f9287d"
Jul 17 13:11:14.629224 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:11:14.628943    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:11:18.629130 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:18.629089    2569 scope.go:117] "RemoveContainer" containerID="1605838e4ede9ca3778943cec2d98b0e7e5e941d9b36252b8210745a8ff7fb5f"
Jul 17 13:11:19.291442 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:19.291412    2569 generic.go:356] "Generic (PLEG): container finished" podID="21ce8b8b-ce9c-4de4-93bf-dd0c052385c6" containerID="409e3f71377b4071a9632d222fca97391bbe054253257cca2b21e23d99ca1a60" exitCode=6
Jul 17 13:11:19.291586 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:19.291494    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8" event={"ID":"21ce8b8b-ce9c-4de4-93bf-dd0c052385c6","Type":"ContainerDied","Data":"409e3f71377b4071a9632d222fca97391bbe054253257cca2b21e23d99ca1a60"}
Jul 17 13:11:19.291586 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:19.291579    2569 scope.go:117] "RemoveContainer" containerID="1605838e4ede9ca3778943cec2d98b0e7e5e941d9b36252b8210745a8ff7fb5f"
Jul 17 13:11:19.291912 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:19.291895    2569 scope.go:117] "RemoveContainer" containerID="409e3f71377b4071a9632d222fca97391bbe054253257cca2b21e23d99ca1a60"
Jul 17 13:11:19.292102 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:11:19.292080    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8_odh-ai-gateway-infra(21ce8b8b-ce9c-4de4-93bf-dd0c052385c6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8" podUID="21ce8b8b-ce9c-4de4-93bf-dd0c052385c6"
Jul 17 13:11:20.328009 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:20.327977    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8"]
Jul 17 13:11:20.460017 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:20.459993    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8"
Jul 17 13:11:20.516574 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:20.516543    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ce8b8b-ce9c-4de4-93bf-dd0c052385c6-maas-api-service-ca\") pod \"21ce8b8b-ce9c-4de4-93bf-dd0c052385c6\" (UID: \"21ce8b8b-ce9c-4de4-93bf-dd0c052385c6\") "
Jul 17 13:11:20.516977 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:20.516958    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ce8b8b-ce9c-4de4-93bf-dd0c052385c6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "21ce8b8b-ce9c-4de4-93bf-dd0c052385c6" (UID: "21ce8b8b-ce9c-4de4-93bf-dd0c052385c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:11:20.617783 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:20.617704    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ce8b8b-ce9c-4de4-93bf-dd0c052385c6-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:11:21.304960 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:21.304932    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8"
Jul 17 13:11:21.305078 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:21.304966    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8" event={"ID":"21ce8b8b-ce9c-4de4-93bf-dd0c052385c6","Type":"ContainerDied","Data":"bc76e68661afa8a0b71bf4b7c6538dd5be7cfdbd9ac43934a342b799bfc1391e"}
Jul 17 13:11:21.305078 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:21.305044    2569 scope.go:117] "RemoveContainer" containerID="409e3f71377b4071a9632d222fca97391bbe054253257cca2b21e23d99ca1a60"
Jul 17 13:11:21.326765 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:21.326730    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8"]
Jul 17 13:11:21.330488 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:21.330464    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zn4m8"]
Jul 17 13:11:21.633783 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:21.633707    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21ce8b8b-ce9c-4de4-93bf-dd0c052385c6" path="/var/lib/kubelet/pods/21ce8b8b-ce9c-4de4-93bf-dd0c052385c6/volumes"
Jul 17 13:11:28.628701 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:28.628622    2569 scope.go:117] "RemoveContainer" containerID="401a7ae3df3aa2bf10592879840e334cccd00cdc103da71b62715d1859f9287d"
Jul 17 13:11:28.629204 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:11:28.628810    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:11:41.749999 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.749965    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7"]
Jul 17 13:11:41.750468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.750276    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ce8b8b-ce9c-4de4-93bf-dd0c052385c6" containerName="revoke-keys"
Jul 17 13:11:41.750468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.750287    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ce8b8b-ce9c-4de4-93bf-dd0c052385c6" containerName="revoke-keys"
Jul 17 13:11:41.750468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.750307    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51796b11-3bdc-4aff-bca5-02182bccae85" containerName="revoke-keys"
Jul 17 13:11:41.750468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.750312    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51796b11-3bdc-4aff-bca5-02182bccae85" containerName="revoke-keys"
Jul 17 13:11:41.750468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.750323    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51796b11-3bdc-4aff-bca5-02182bccae85" containerName="revoke-keys"
Jul 17 13:11:41.750468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.750329    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51796b11-3bdc-4aff-bca5-02182bccae85" containerName="revoke-keys"
Jul 17 13:11:41.750468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.750335    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ce8b8b-ce9c-4de4-93bf-dd0c052385c6" containerName="revoke-keys"
Jul 17 13:11:41.750468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.750341    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ce8b8b-ce9c-4de4-93bf-dd0c052385c6" containerName="revoke-keys"
Jul 17 13:11:41.750468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.750347    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ce8b8b-ce9c-4de4-93bf-dd0c052385c6" containerName="revoke-keys"
Jul 17 13:11:41.750468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.750351    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ce8b8b-ce9c-4de4-93bf-dd0c052385c6" containerName="revoke-keys"
Jul 17 13:11:41.750468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.750361    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51796b11-3bdc-4aff-bca5-02182bccae85" containerName="revoke-keys"
Jul 17 13:11:41.750468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.750365    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="51796b11-3bdc-4aff-bca5-02182bccae85" containerName="revoke-keys"
Jul 17 13:11:41.750468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.750414    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="51796b11-3bdc-4aff-bca5-02182bccae85" containerName="revoke-keys"
Jul 17 13:11:41.750468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.750422    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="21ce8b8b-ce9c-4de4-93bf-dd0c052385c6" containerName="revoke-keys"
Jul 17 13:11:41.750468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.750428    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="51796b11-3bdc-4aff-bca5-02182bccae85" containerName="revoke-keys"
Jul 17 13:11:41.750468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.750433    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="21ce8b8b-ce9c-4de4-93bf-dd0c052385c6" containerName="revoke-keys"
Jul 17 13:11:41.750468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.750439    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="21ce8b8b-ce9c-4de4-93bf-dd0c052385c6" containerName="revoke-keys"
Jul 17 13:11:41.750468 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.750445    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="51796b11-3bdc-4aff-bca5-02182bccae85" containerName="revoke-keys"
Jul 17 13:11:41.754563 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.754544    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7"
Jul 17 13:11:41.757357 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.757329    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:11:41.761572 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.761546    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7"]
Jul 17 13:11:41.869341 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.869309    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7\" (UID: \"a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7"
Jul 17 13:11:41.970411 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.970385    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7\" (UID: \"a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7"
Jul 17 13:11:41.970986 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:41.970967    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7\" (UID: \"a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7"
Jul 17 13:11:42.065793 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:42.065728    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7"
Jul 17 13:11:42.197711 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:42.197685    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7"]
Jul 17 13:11:42.202940 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:11:42.202917    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22d8a9a_d121_4a5c_ba7f_6b338aa3cf53.slice/crio-73153b7b4ab0af1d3128ec6ddcf1daa33cd5953b51195a0a7c7789333ba0c9e9 WatchSource:0}: Error finding container 73153b7b4ab0af1d3128ec6ddcf1daa33cd5953b51195a0a7c7789333ba0c9e9: Status 404 returned error can't find the container with id 73153b7b4ab0af1d3128ec6ddcf1daa33cd5953b51195a0a7c7789333ba0c9e9
Jul 17 13:11:42.408164 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:42.408105    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7" event={"ID":"a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53","Type":"ContainerStarted","Data":"73153b7b4ab0af1d3128ec6ddcf1daa33cd5953b51195a0a7c7789333ba0c9e9"}
Jul 17 13:11:43.414935 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:43.414902    2569 generic.go:356] "Generic (PLEG): container finished" podID="a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53" containerID="8f00c6350e622c9ce27e3d69df91c08b0747dbfe5b30ff69b0159566bb5ab087" exitCode=6
Jul 17 13:11:43.415281 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:43.414962    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7" event={"ID":"a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53","Type":"ContainerDied","Data":"8f00c6350e622c9ce27e3d69df91c08b0747dbfe5b30ff69b0159566bb5ab087"}
Jul 17 13:11:43.415281 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:43.415177    2569 scope.go:117] "RemoveContainer" containerID="8f00c6350e622c9ce27e3d69df91c08b0747dbfe5b30ff69b0159566bb5ab087"
Jul 17 13:11:43.628868 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:43.628834    2569 scope.go:117] "RemoveContainer" containerID="401a7ae3df3aa2bf10592879840e334cccd00cdc103da71b62715d1859f9287d"
Jul 17 13:11:43.629041 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:11:43.629021    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:11:44.420980 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:44.420910    2569 generic.go:356] "Generic (PLEG): container finished" podID="a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53" containerID="4874494f55b6c8965d6979c83c13c626f87d5697ad3486feab78734606bd7b99" exitCode=6
Jul 17 13:11:44.421348 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:44.420992    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7" event={"ID":"a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53","Type":"ContainerDied","Data":"4874494f55b6c8965d6979c83c13c626f87d5697ad3486feab78734606bd7b99"}
Jul 17 13:11:44.421348 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:44.421051    2569 scope.go:117] "RemoveContainer" containerID="8f00c6350e622c9ce27e3d69df91c08b0747dbfe5b30ff69b0159566bb5ab087"
Jul 17 13:11:44.421348 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:44.421191    2569 scope.go:117] "RemoveContainer" containerID="4874494f55b6c8965d6979c83c13c626f87d5697ad3486feab78734606bd7b99"
Jul 17 13:11:44.421504 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:11:44.421373    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7_odh-ai-gateway-infra(a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7" podUID="a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53"
Jul 17 13:11:45.428041 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:45.427975    2569 scope.go:117] "RemoveContainer" containerID="4874494f55b6c8965d6979c83c13c626f87d5697ad3486feab78734606bd7b99"
Jul 17 13:11:45.428360 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:11:45.428136    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7_odh-ai-gateway-infra(a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7" podUID="a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53"
Jul 17 13:11:57.628353 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:57.628320    2569 scope.go:117] "RemoveContainer" containerID="401a7ae3df3aa2bf10592879840e334cccd00cdc103da71b62715d1859f9287d"
Jul 17 13:11:57.628829 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:11:57.628534    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:11:58.628359 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:58.628326    2569 scope.go:117] "RemoveContainer" containerID="4874494f55b6c8965d6979c83c13c626f87d5697ad3486feab78734606bd7b99"
Jul 17 13:11:59.494944 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:59.494910    2569 generic.go:356] "Generic (PLEG): container finished" podID="a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53" containerID="d26dd9a778b0b567cb77f929133ef69485082f2c3aafdc85763e166ae779c4ab" exitCode=6
Jul 17 13:11:59.495065 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:59.494995    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7" event={"ID":"a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53","Type":"ContainerDied","Data":"d26dd9a778b0b567cb77f929133ef69485082f2c3aafdc85763e166ae779c4ab"}
Jul 17 13:11:59.495065 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:59.495055    2569 scope.go:117] "RemoveContainer" containerID="4874494f55b6c8965d6979c83c13c626f87d5697ad3486feab78734606bd7b99"
Jul 17 13:11:59.495331 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:11:59.495315    2569 scope.go:117] "RemoveContainer" containerID="d26dd9a778b0b567cb77f929133ef69485082f2c3aafdc85763e166ae779c4ab"
Jul 17 13:11:59.495522 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:11:59.495501    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7_odh-ai-gateway-infra(a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7" podUID="a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53"
Jul 17 13:12:00.525472 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:00.525430    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7"]
Jul 17 13:12:00.595513 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:00.595480    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z"]
Jul 17 13:12:00.600069 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:00.600051    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z"
Jul 17 13:12:00.609364 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:00.609331    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z"]
Jul 17 13:12:00.662855 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:00.662836    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7"
Jul 17 13:12:00.704939 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:00.704912    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53-maas-api-service-ca\") pod \"a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53\" (UID: \"a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53\") "
Jul 17 13:12:00.705240 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:00.705220    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53" (UID: "a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:12:00.705295 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:00.705239    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e1b809c-aa1e-4f4c-902e-112203215385-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z\" (UID: \"3e1b809c-aa1e-4f4c-902e-112203215385\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z"
Jul 17 13:12:00.705295 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:00.705287    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:12:00.806472 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:00.806415    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e1b809c-aa1e-4f4c-902e-112203215385-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z\" (UID: \"3e1b809c-aa1e-4f4c-902e-112203215385\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z"
Jul 17 13:12:00.806979 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:00.806963    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e1b809c-aa1e-4f4c-902e-112203215385-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z\" (UID: \"3e1b809c-aa1e-4f4c-902e-112203215385\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z"
Jul 17 13:12:00.911905 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:00.911881    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z"
Jul 17 13:12:01.038160 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:01.038100    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z"]
Jul 17 13:12:01.040091 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:12:01.040058    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e1b809c_aa1e_4f4c_902e_112203215385.slice/crio-816651345ad370e37aade87459bbd9e8ec7c5d54e6613822258706680a6ee977 WatchSource:0}: Error finding container 816651345ad370e37aade87459bbd9e8ec7c5d54e6613822258706680a6ee977: Status 404 returned error can't find the container with id 816651345ad370e37aade87459bbd9e8ec7c5d54e6613822258706680a6ee977
Jul 17 13:12:01.507323 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:01.507296    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7"
Jul 17 13:12:01.507454 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:01.507330    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7" event={"ID":"a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53","Type":"ContainerDied","Data":"73153b7b4ab0af1d3128ec6ddcf1daa33cd5953b51195a0a7c7789333ba0c9e9"}
Jul 17 13:12:01.507454 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:01.507394    2569 scope.go:117] "RemoveContainer" containerID="d26dd9a778b0b567cb77f929133ef69485082f2c3aafdc85763e166ae779c4ab"
Jul 17 13:12:01.508923 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:01.508895    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z" event={"ID":"3e1b809c-aa1e-4f4c-902e-112203215385","Type":"ContainerStarted","Data":"9a322ba89712ccb3ce2b60557b087d6e8aa0d07302a18b7b3623384a4533875e"}
Jul 17 13:12:01.509022 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:01.508940    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z" event={"ID":"3e1b809c-aa1e-4f4c-902e-112203215385","Type":"ContainerStarted","Data":"816651345ad370e37aade87459bbd9e8ec7c5d54e6613822258706680a6ee977"}
Jul 17 13:12:01.528224 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:01.528171    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z" podStartSLOduration=1.5281427380000001 podStartE2EDuration="1.528142738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:12:01.528111722 +0000 UTC m=+2996.532158606" watchObservedRunningTime="2026-07-17 13:12:01.528142738 +0000 UTC m=+2996.532189623"
Jul 17 13:12:01.542485 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:01.542464    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7"]
Jul 17 13:12:01.551383 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:01.551355    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-h9hl7"]
Jul 17 13:12:01.634039 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:01.634016    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53" path="/var/lib/kubelet/pods/a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53/volumes"
Jul 17 13:12:02.515740 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:02.515711    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e1b809c-aa1e-4f4c-902e-112203215385" containerID="9a322ba89712ccb3ce2b60557b087d6e8aa0d07302a18b7b3623384a4533875e" exitCode=6
Jul 17 13:12:02.515920 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:02.515783    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z" event={"ID":"3e1b809c-aa1e-4f4c-902e-112203215385","Type":"ContainerDied","Data":"9a322ba89712ccb3ce2b60557b087d6e8aa0d07302a18b7b3623384a4533875e"}
Jul 17 13:12:02.516023 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:02.516005    2569 scope.go:117] "RemoveContainer" containerID="9a322ba89712ccb3ce2b60557b087d6e8aa0d07302a18b7b3623384a4533875e"
Jul 17 13:12:03.150789 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:12:03.150765    2569 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e1b809c_aa1e_4f4c_902e_112203215385.slice/crio-daba0ea14381c28d6e78352884dc683c0a75a508f0cd49b3487765cdc9c8554d.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 13:12:03.522707 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:03.522677    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e1b809c-aa1e-4f4c-902e-112203215385" containerID="daba0ea14381c28d6e78352884dc683c0a75a508f0cd49b3487765cdc9c8554d" exitCode=6
Jul 17 13:12:03.522852 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:03.522790    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z" event={"ID":"3e1b809c-aa1e-4f4c-902e-112203215385","Type":"ContainerDied","Data":"daba0ea14381c28d6e78352884dc683c0a75a508f0cd49b3487765cdc9c8554d"}
Jul 17 13:12:03.522902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:03.522852    2569 scope.go:117] "RemoveContainer" containerID="9a322ba89712ccb3ce2b60557b087d6e8aa0d07302a18b7b3623384a4533875e"
Jul 17 13:12:03.523055 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:03.523035    2569 scope.go:117] "RemoveContainer" containerID="daba0ea14381c28d6e78352884dc683c0a75a508f0cd49b3487765cdc9c8554d"
Jul 17 13:12:03.523271 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:12:03.523240    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z_odh-ai-gateway-infra(3e1b809c-aa1e-4f4c-902e-112203215385)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z" podUID="3e1b809c-aa1e-4f4c-902e-112203215385"
Jul 17 13:12:04.529542 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:04.529513    2569 scope.go:117] "RemoveContainer" containerID="daba0ea14381c28d6e78352884dc683c0a75a508f0cd49b3487765cdc9c8554d"
Jul 17 13:12:04.529976 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:12:04.529728    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z_odh-ai-gateway-infra(3e1b809c-aa1e-4f4c-902e-112203215385)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z" podUID="3e1b809c-aa1e-4f4c-902e-112203215385"
Jul 17 13:12:12.629007 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:12.628977    2569 scope.go:117] "RemoveContainer" containerID="401a7ae3df3aa2bf10592879840e334cccd00cdc103da71b62715d1859f9287d"
Jul 17 13:12:12.629443 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:12:12.629137    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:12:13.435112 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:13.434996    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 13:12:13.439320 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:13.439299    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 13:12:16.628299 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:16.628269    2569 scope.go:117] "RemoveContainer" containerID="daba0ea14381c28d6e78352884dc683c0a75a508f0cd49b3487765cdc9c8554d"
Jul 17 13:12:17.589914 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:17.589846    2569 generic.go:356] "Generic (PLEG): container finished" podID="3e1b809c-aa1e-4f4c-902e-112203215385" containerID="7446e99a9cae84c7c4873cc61a9d457a243b12cf9820223fa06859d6ab1a8285" exitCode=6
Jul 17 13:12:17.589914 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:17.589896    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z" event={"ID":"3e1b809c-aa1e-4f4c-902e-112203215385","Type":"ContainerDied","Data":"7446e99a9cae84c7c4873cc61a9d457a243b12cf9820223fa06859d6ab1a8285"}
Jul 17 13:12:17.590106 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:17.589945    2569 scope.go:117] "RemoveContainer" containerID="daba0ea14381c28d6e78352884dc683c0a75a508f0cd49b3487765cdc9c8554d"
Jul 17 13:12:17.590249 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:17.590229    2569 scope.go:117] "RemoveContainer" containerID="7446e99a9cae84c7c4873cc61a9d457a243b12cf9820223fa06859d6ab1a8285"
Jul 17 13:12:17.590449 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:12:17.590427    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z_odh-ai-gateway-infra(3e1b809c-aa1e-4f4c-902e-112203215385)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z" podUID="3e1b809c-aa1e-4f4c-902e-112203215385"
Jul 17 13:12:18.616899 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:18.616856    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z"]
Jul 17 13:12:18.750854 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:18.750832    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z"
Jul 17 13:12:18.843538 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:18.843512    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e1b809c-aa1e-4f4c-902e-112203215385-maas-api-service-ca\") pod \"3e1b809c-aa1e-4f4c-902e-112203215385\" (UID: \"3e1b809c-aa1e-4f4c-902e-112203215385\") "
Jul 17 13:12:18.843881 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:18.843856    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e1b809c-aa1e-4f4c-902e-112203215385-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3e1b809c-aa1e-4f4c-902e-112203215385" (UID: "3e1b809c-aa1e-4f4c-902e-112203215385"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:12:18.945062 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:18.944998    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e1b809c-aa1e-4f4c-902e-112203215385-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:12:19.602874 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:19.602837    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z" event={"ID":"3e1b809c-aa1e-4f4c-902e-112203215385","Type":"ContainerDied","Data":"816651345ad370e37aade87459bbd9e8ec7c5d54e6613822258706680a6ee977"}
Jul 17 13:12:19.602998 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:19.602921    2569 scope.go:117] "RemoveContainer" containerID="7446e99a9cae84c7c4873cc61a9d457a243b12cf9820223fa06859d6ab1a8285"
Jul 17 13:12:19.602998 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:19.602849    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z"
Jul 17 13:12:19.625044 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:19.625017    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z"]
Jul 17 13:12:19.634021 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:19.633998    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-rdf7z"]
Jul 17 13:12:21.634426 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:21.634394    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e1b809c-aa1e-4f4c-902e-112203215385" path="/var/lib/kubelet/pods/3e1b809c-aa1e-4f4c-902e-112203215385/volumes"
Jul 17 13:12:27.628901 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:27.628867    2569 scope.go:117] "RemoveContainer" containerID="401a7ae3df3aa2bf10592879840e334cccd00cdc103da71b62715d1859f9287d"
Jul 17 13:12:27.629286 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:12:27.629042    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:12:31.802005 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:31.801976    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz"]
Jul 17 13:12:31.802406 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:31.802280    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53" containerName="revoke-keys"
Jul 17 13:12:31.802406 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:31.802291    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53" containerName="revoke-keys"
Jul 17 13:12:31.802406 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:31.802299    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e1b809c-aa1e-4f4c-902e-112203215385" containerName="revoke-keys"
Jul 17 13:12:31.802406 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:31.802306    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e1b809c-aa1e-4f4c-902e-112203215385" containerName="revoke-keys"
Jul 17 13:12:31.802406 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:31.802320    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53" containerName="revoke-keys"
Jul 17 13:12:31.802406 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:31.802327    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53" containerName="revoke-keys"
Jul 17 13:12:31.802406 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:31.802333    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e1b809c-aa1e-4f4c-902e-112203215385" containerName="revoke-keys"
Jul 17 13:12:31.802406 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:31.802339    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e1b809c-aa1e-4f4c-902e-112203215385" containerName="revoke-keys"
Jul 17 13:12:31.802406 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:31.802397    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53" containerName="revoke-keys"
Jul 17 13:12:31.802406 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:31.802403    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e1b809c-aa1e-4f4c-902e-112203215385" containerName="revoke-keys"
Jul 17 13:12:31.802406 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:31.802410    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e1b809c-aa1e-4f4c-902e-112203215385" containerName="revoke-keys"
Jul 17 13:12:31.802838 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:31.802417    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53" containerName="revoke-keys"
Jul 17 13:12:31.802838 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:31.802423    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53" containerName="revoke-keys"
Jul 17 13:12:31.806862 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:31.806835    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz"
Jul 17 13:12:31.811197 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:31.811175    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:12:31.816115 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:31.816083    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz"]
Jul 17 13:12:31.942632 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:31.942594    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34255402-5b50-4c99-918d-ed3f0711a238-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz\" (UID: \"34255402-5b50-4c99-918d-ed3f0711a238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz"
Jul 17 13:12:32.042982 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:32.042961    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34255402-5b50-4c99-918d-ed3f0711a238-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz\" (UID: \"34255402-5b50-4c99-918d-ed3f0711a238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz"
Jul 17 13:12:32.043497 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:32.043479    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34255402-5b50-4c99-918d-ed3f0711a238-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz\" (UID: \"34255402-5b50-4c99-918d-ed3f0711a238\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz"
Jul 17 13:12:32.118667 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:32.118619    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz"
Jul 17 13:12:32.467034 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:32.467011    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz"]
Jul 17 13:12:32.469477 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:12:32.469450    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34255402_5b50_4c99_918d_ed3f0711a238.slice/crio-76b1377fc563d87dbf6b4f85ab9d112ba998894461c4324f0a8a9760dc052d43 WatchSource:0}: Error finding container 76b1377fc563d87dbf6b4f85ab9d112ba998894461c4324f0a8a9760dc052d43: Status 404 returned error can't find the container with id 76b1377fc563d87dbf6b4f85ab9d112ba998894461c4324f0a8a9760dc052d43
Jul 17 13:12:32.663854 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:32.663823    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz" event={"ID":"34255402-5b50-4c99-918d-ed3f0711a238","Type":"ContainerStarted","Data":"9e25affae7436d35e5f74d533e000d988c679751c785232371613a327488a8ec"}
Jul 17 13:12:32.663967 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:32.663887    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz" event={"ID":"34255402-5b50-4c99-918d-ed3f0711a238","Type":"ContainerStarted","Data":"76b1377fc563d87dbf6b4f85ab9d112ba998894461c4324f0a8a9760dc052d43"}
Jul 17 13:12:32.680532 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:32.680489    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz" podStartSLOduration=1.680474594 podStartE2EDuration="1.680474594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:12:32.679537585 +0000 UTC m=+3027.683584469" watchObservedRunningTime="2026-07-17 13:12:32.680474594 +0000 UTC m=+3027.684521476"
Jul 17 13:12:33.670040 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:33.670008    2569 generic.go:356] "Generic (PLEG): container finished" podID="34255402-5b50-4c99-918d-ed3f0711a238" containerID="9e25affae7436d35e5f74d533e000d988c679751c785232371613a327488a8ec" exitCode=6
Jul 17 13:12:33.670476 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:33.670093    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz" event={"ID":"34255402-5b50-4c99-918d-ed3f0711a238","Type":"ContainerDied","Data":"9e25affae7436d35e5f74d533e000d988c679751c785232371613a327488a8ec"}
Jul 17 13:12:33.670476 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:33.670251    2569 scope.go:117] "RemoveContainer" containerID="9e25affae7436d35e5f74d533e000d988c679751c785232371613a327488a8ec"
Jul 17 13:12:34.676804 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:34.676768    2569 generic.go:356] "Generic (PLEG): container finished" podID="34255402-5b50-4c99-918d-ed3f0711a238" containerID="7383fa319ea507463f3a71483038ae531661574be953ef2c9cf79eabafbd5a5b" exitCode=6
Jul 17 13:12:34.677241 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:34.676839    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz" event={"ID":"34255402-5b50-4c99-918d-ed3f0711a238","Type":"ContainerDied","Data":"7383fa319ea507463f3a71483038ae531661574be953ef2c9cf79eabafbd5a5b"}
Jul 17 13:12:34.677241 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:34.676909    2569 scope.go:117] "RemoveContainer" containerID="9e25affae7436d35e5f74d533e000d988c679751c785232371613a327488a8ec"
Jul 17 13:12:34.677241 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:34.677069    2569 scope.go:117] "RemoveContainer" containerID="7383fa319ea507463f3a71483038ae531661574be953ef2c9cf79eabafbd5a5b"
Jul 17 13:12:34.677413 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:12:34.677290    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz_odh-ai-gateway-infra(34255402-5b50-4c99-918d-ed3f0711a238)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz" podUID="34255402-5b50-4c99-918d-ed3f0711a238"
Jul 17 13:12:35.682685 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:35.682658    2569 scope.go:117] "RemoveContainer" containerID="7383fa319ea507463f3a71483038ae531661574be953ef2c9cf79eabafbd5a5b"
Jul 17 13:12:35.683109 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:12:35.682846    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz_odh-ai-gateway-infra(34255402-5b50-4c99-918d-ed3f0711a238)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz" podUID="34255402-5b50-4c99-918d-ed3f0711a238"
Jul 17 13:12:38.628794 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:38.628736    2569 scope.go:117] "RemoveContainer" containerID="401a7ae3df3aa2bf10592879840e334cccd00cdc103da71b62715d1859f9287d"
Jul 17 13:12:38.629189 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:12:38.628951    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:12:50.628269 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:50.628232    2569 scope.go:117] "RemoveContainer" containerID="7383fa319ea507463f3a71483038ae531661574be953ef2c9cf79eabafbd5a5b"
Jul 17 13:12:51.756682 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:51.756653    2569 generic.go:356] "Generic (PLEG): container finished" podID="34255402-5b50-4c99-918d-ed3f0711a238" containerID="da720391cced36e79bd525a053a626fea8bb742f5364852aefc52f8afe50e2ae" exitCode=6
Jul 17 13:12:51.757152 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:51.756729    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz" event={"ID":"34255402-5b50-4c99-918d-ed3f0711a238","Type":"ContainerDied","Data":"da720391cced36e79bd525a053a626fea8bb742f5364852aefc52f8afe50e2ae"}
Jul 17 13:12:51.757152 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:51.756813    2569 scope.go:117] "RemoveContainer" containerID="7383fa319ea507463f3a71483038ae531661574be953ef2c9cf79eabafbd5a5b"
Jul 17 13:12:51.757152 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:51.757110    2569 scope.go:117] "RemoveContainer" containerID="da720391cced36e79bd525a053a626fea8bb742f5364852aefc52f8afe50e2ae"
Jul 17 13:12:51.757327 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:12:51.757310    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz_odh-ai-gateway-infra(34255402-5b50-4c99-918d-ed3f0711a238)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz" podUID="34255402-5b50-4c99-918d-ed3f0711a238"
Jul 17 13:12:52.786421 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:52.786390    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz"]
Jul 17 13:12:52.912561 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:52.912541    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz"
Jul 17 13:12:53.093582 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:53.093518    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34255402-5b50-4c99-918d-ed3f0711a238-maas-api-service-ca\") pod \"34255402-5b50-4c99-918d-ed3f0711a238\" (UID: \"34255402-5b50-4c99-918d-ed3f0711a238\") "
Jul 17 13:12:53.093911 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:53.093892    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34255402-5b50-4c99-918d-ed3f0711a238-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "34255402-5b50-4c99-918d-ed3f0711a238" (UID: "34255402-5b50-4c99-918d-ed3f0711a238"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:12:53.194733 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:53.194708    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34255402-5b50-4c99-918d-ed3f0711a238-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:12:53.628619 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:53.628594    2569 scope.go:117] "RemoveContainer" containerID="401a7ae3df3aa2bf10592879840e334cccd00cdc103da71b62715d1859f9287d"
Jul 17 13:12:53.628791 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:12:53.628746    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:12:53.768101 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:53.768078    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz"
Jul 17 13:12:53.768101 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:53.768088    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz" event={"ID":"34255402-5b50-4c99-918d-ed3f0711a238","Type":"ContainerDied","Data":"76b1377fc563d87dbf6b4f85ab9d112ba998894461c4324f0a8a9760dc052d43"}
Jul 17 13:12:53.768293 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:53.768137    2569 scope.go:117] "RemoveContainer" containerID="da720391cced36e79bd525a053a626fea8bb742f5364852aefc52f8afe50e2ae"
Jul 17 13:12:53.786044 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:53.785992    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz"]
Jul 17 13:12:53.787957 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:53.787935    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-d7ghz"]
Jul 17 13:12:55.639829 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:12:55.639795    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34255402-5b50-4c99-918d-ed3f0711a238" path="/var/lib/kubelet/pods/34255402-5b50-4c99-918d-ed3f0711a238/volumes"
Jul 17 13:13:06.629084 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:06.629055    2569 scope.go:117] "RemoveContainer" containerID="401a7ae3df3aa2bf10592879840e334cccd00cdc103da71b62715d1859f9287d"
Jul 17 13:13:06.629447 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:06.629345    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 13:13:07.832184 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:07.832121    2569 generic.go:356] "Generic (PLEG): container finished" podID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerID="c761b66d2907cdf42b88abd18950c08e483ce23b9a5298f086fda10d90d2f980" exitCode=1
Jul 17 13:13:07.832184 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:07.832157    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" event={"ID":"bcff1798-e8d8-4449-b8e3-eaf723b636c4","Type":"ContainerDied","Data":"c761b66d2907cdf42b88abd18950c08e483ce23b9a5298f086fda10d90d2f980"}
Jul 17 13:13:07.832549 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:07.832223    2569 scope.go:117] "RemoveContainer" containerID="401a7ae3df3aa2bf10592879840e334cccd00cdc103da71b62715d1859f9287d"
Jul 17 13:13:07.832549 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:07.832502    2569 scope.go:117] "RemoveContainer" containerID="c761b66d2907cdf42b88abd18950c08e483ce23b9a5298f086fda10d90d2f980"
Jul 17 13:13:07.832707 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:07.832683    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:13:09.493621 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.493588    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw"]
Jul 17 13:13:09.494036 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.493936    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34255402-5b50-4c99-918d-ed3f0711a238" containerName="revoke-keys"
Jul 17 13:13:09.494036 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.493949    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="34255402-5b50-4c99-918d-ed3f0711a238" containerName="revoke-keys"
Jul 17 13:13:09.494036 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.493965    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53" containerName="revoke-keys"
Jul 17 13:13:09.494036 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.493970    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a22d8a9a-d121-4a5c-ba7f-6b338aa3cf53" containerName="revoke-keys"
Jul 17 13:13:09.494036 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.493977    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34255402-5b50-4c99-918d-ed3f0711a238" containerName="revoke-keys"
Jul 17 13:13:09.494036 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.493983    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="34255402-5b50-4c99-918d-ed3f0711a238" containerName="revoke-keys"
Jul 17 13:13:09.494036 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.493990    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34255402-5b50-4c99-918d-ed3f0711a238" containerName="revoke-keys"
Jul 17 13:13:09.494036 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.493995    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="34255402-5b50-4c99-918d-ed3f0711a238" containerName="revoke-keys"
Jul 17 13:13:09.494036 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.494007    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e1b809c-aa1e-4f4c-902e-112203215385" containerName="revoke-keys"
Jul 17 13:13:09.494036 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.494012    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e1b809c-aa1e-4f4c-902e-112203215385" containerName="revoke-keys"
Jul 17 13:13:09.494372 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.494072    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="34255402-5b50-4c99-918d-ed3f0711a238" containerName="revoke-keys"
Jul 17 13:13:09.494372 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.494079    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e1b809c-aa1e-4f4c-902e-112203215385" containerName="revoke-keys"
Jul 17 13:13:09.494372 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.494088    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="34255402-5b50-4c99-918d-ed3f0711a238" containerName="revoke-keys"
Jul 17 13:13:09.498241 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.498222    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw"
Jul 17 13:13:09.501098 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.501079    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:13:09.504661 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.504635    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw"]
Jul 17 13:13:09.510133 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.510100    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb02592-f3cb-4062-bfbc-f3e3f68c5656-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw\" (UID: \"fbb02592-f3cb-4062-bfbc-f3e3f68c5656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw"
Jul 17 13:13:09.610895 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.610868    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb02592-f3cb-4062-bfbc-f3e3f68c5656-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw\" (UID: \"fbb02592-f3cb-4062-bfbc-f3e3f68c5656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw"
Jul 17 13:13:09.611441 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.611421    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb02592-f3cb-4062-bfbc-f3e3f68c5656-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw\" (UID: \"fbb02592-f3cb-4062-bfbc-f3e3f68c5656\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw"
Jul 17 13:13:09.811501 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.811442    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw"
Jul 17 13:13:09.943558 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:09.943536    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw"]
Jul 17 13:13:09.945538 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:13:09.945503    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb02592_f3cb_4062_bfbc_f3e3f68c5656.slice/crio-9e88482539125bdfda83ee6383680b9913cb512af856ed0763eb455de2682721 WatchSource:0}: Error finding container 9e88482539125bdfda83ee6383680b9913cb512af856ed0763eb455de2682721: Status 404 returned error can't find the container with id 9e88482539125bdfda83ee6383680b9913cb512af856ed0763eb455de2682721
Jul 17 13:13:10.848405 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:10.848346    2569 generic.go:356] "Generic (PLEG): container finished" podID="fbb02592-f3cb-4062-bfbc-f3e3f68c5656" containerID="4613a17b7388fb7e7cbe98ac25af9ccd251329012ac90976cd2439c25ec1d485" exitCode=7
Jul 17 13:13:10.848405 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:10.848379    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw" event={"ID":"fbb02592-f3cb-4062-bfbc-f3e3f68c5656","Type":"ContainerDied","Data":"4613a17b7388fb7e7cbe98ac25af9ccd251329012ac90976cd2439c25ec1d485"}
Jul 17 13:13:10.848745 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:10.848417    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw" event={"ID":"fbb02592-f3cb-4062-bfbc-f3e3f68c5656","Type":"ContainerStarted","Data":"9e88482539125bdfda83ee6383680b9913cb512af856ed0763eb455de2682721"}
Jul 17 13:13:10.848745 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:10.848565    2569 scope.go:117] "RemoveContainer" containerID="4613a17b7388fb7e7cbe98ac25af9ccd251329012ac90976cd2439c25ec1d485"
Jul 17 13:13:11.271388 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:11.271350    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:13:11.271727 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:11.271710    2569 scope.go:117] "RemoveContainer" containerID="c761b66d2907cdf42b88abd18950c08e483ce23b9a5298f086fda10d90d2f980"
Jul 17 13:13:11.271937 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:11.271914    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:13:11.854055 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:11.854029    2569 generic.go:356] "Generic (PLEG): container finished" podID="fbb02592-f3cb-4062-bfbc-f3e3f68c5656" containerID="19c219c385c6d15f4a8a817a43fcbc9d8a8088858ea992d63d275708a77f140b" exitCode=7
Jul 17 13:13:11.854400 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:11.854117    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw" event={"ID":"fbb02592-f3cb-4062-bfbc-f3e3f68c5656","Type":"ContainerDied","Data":"19c219c385c6d15f4a8a817a43fcbc9d8a8088858ea992d63d275708a77f140b"}
Jul 17 13:13:11.854400 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:11.854171    2569 scope.go:117] "RemoveContainer" containerID="4613a17b7388fb7e7cbe98ac25af9ccd251329012ac90976cd2439c25ec1d485"
Jul 17 13:13:11.854400 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:11.854278    2569 scope.go:117] "RemoveContainer" containerID="19c219c385c6d15f4a8a817a43fcbc9d8a8088858ea992d63d275708a77f140b"
Jul 17 13:13:11.854562 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:11.854463    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw_odh-ai-gateway-infra(fbb02592-f3cb-4062-bfbc-f3e3f68c5656)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw" podUID="fbb02592-f3cb-4062-bfbc-f3e3f68c5656"
Jul 17 13:13:12.860543 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:12.860514    2569 scope.go:117] "RemoveContainer" containerID="19c219c385c6d15f4a8a817a43fcbc9d8a8088858ea992d63d275708a77f140b"
Jul 17 13:13:12.861024 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:12.860730    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw_odh-ai-gateway-infra(fbb02592-f3cb-4062-bfbc-f3e3f68c5656)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw" podUID="fbb02592-f3cb-4062-bfbc-f3e3f68c5656"
Jul 17 13:13:15.389230 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:15.389193    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:13:15.389659 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:15.389562    2569 scope.go:117] "RemoveContainer" containerID="c761b66d2907cdf42b88abd18950c08e483ce23b9a5298f086fda10d90d2f980"
Jul 17 13:13:15.389802 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:15.389777    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:13:22.383315 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:22.383276    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:13:25.632170 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:25.632138    2569 scope.go:117] "RemoveContainer" containerID="c761b66d2907cdf42b88abd18950c08e483ce23b9a5298f086fda10d90d2f980"
Jul 17 13:13:25.632670 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:25.632365    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5_odh-ai-gateway-infra(bcff1798-e8d8-4449-b8e3-eaf723b636c4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4"
Jul 17 13:13:25.822791 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:25.822747    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"]
Jul 17 13:13:26.046765 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:26.046730    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:13:26.129575 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:26.129550    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcff1798-e8d8-4449-b8e3-eaf723b636c4-maas-api-tls\") pod \"bcff1798-e8d8-4449-b8e3-eaf723b636c4\" (UID: \"bcff1798-e8d8-4449-b8e3-eaf723b636c4\") "
Jul 17 13:13:26.129703 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:26.129589    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8x4z\" (UniqueName: \"kubernetes.io/projected/bcff1798-e8d8-4449-b8e3-eaf723b636c4-kube-api-access-z8x4z\") pod \"bcff1798-e8d8-4449-b8e3-eaf723b636c4\" (UID: \"bcff1798-e8d8-4449-b8e3-eaf723b636c4\") "
Jul 17 13:13:26.129703 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:26.129663    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcff1798-e8d8-4449-b8e3-eaf723b636c4-openshift-service-ca\") pod \"bcff1798-e8d8-4449-b8e3-eaf723b636c4\" (UID: \"bcff1798-e8d8-4449-b8e3-eaf723b636c4\") "
Jul 17 13:13:26.130021 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:26.129996    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcff1798-e8d8-4449-b8e3-eaf723b636c4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bcff1798-e8d8-4449-b8e3-eaf723b636c4" (UID: "bcff1798-e8d8-4449-b8e3-eaf723b636c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:13:26.131514 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:26.131491    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcff1798-e8d8-4449-b8e3-eaf723b636c4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bcff1798-e8d8-4449-b8e3-eaf723b636c4" (UID: "bcff1798-e8d8-4449-b8e3-eaf723b636c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:13:26.131616 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:26.131602    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcff1798-e8d8-4449-b8e3-eaf723b636c4-kube-api-access-z8x4z" (OuterVolumeSpecName: "kube-api-access-z8x4z") pod "bcff1798-e8d8-4449-b8e3-eaf723b636c4" (UID: "bcff1798-e8d8-4449-b8e3-eaf723b636c4"). InnerVolumeSpecName "kube-api-access-z8x4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:13:26.230205 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:26.230186    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcff1798-e8d8-4449-b8e3-eaf723b636c4-maas-api-tls\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:13:26.230287 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:26.230207    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8x4z\" (UniqueName: \"kubernetes.io/projected/bcff1798-e8d8-4449-b8e3-eaf723b636c4-kube-api-access-z8x4z\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:13:26.230287 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:26.230217    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcff1798-e8d8-4449-b8e3-eaf723b636c4-openshift-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:13:26.921836 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:26.921809    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"
Jul 17 13:13:26.922321 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:26.921852    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5" event={"ID":"bcff1798-e8d8-4449-b8e3-eaf723b636c4","Type":"ContainerDied","Data":"c095d0c396f551c25e1baf2b90e5afb3097be597c78c16b0b364f496170b1d85"}
Jul 17 13:13:26.922321 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:26.921917    2569 scope.go:117] "RemoveContainer" containerID="c761b66d2907cdf42b88abd18950c08e483ce23b9a5298f086fda10d90d2f980"
Jul 17 13:13:26.950182 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:26.950159    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"]
Jul 17 13:13:26.956064 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:26.956042    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-12f6c9d7-56d96f899d-fspb5"]
Jul 17 13:13:27.628556 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:27.628526    2569 scope.go:117] "RemoveContainer" containerID="19c219c385c6d15f4a8a817a43fcbc9d8a8088858ea992d63d275708a77f140b"
Jul 17 13:13:27.634293 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:27.634264    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" path="/var/lib/kubelet/pods/bcff1798-e8d8-4449-b8e3-eaf723b636c4/volumes"
Jul 17 13:13:28.935638 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:28.935601    2569 generic.go:356] "Generic (PLEG): container finished" podID="fbb02592-f3cb-4062-bfbc-f3e3f68c5656" containerID="5fe9e2a178cc935ecc755625e9e0a518b13ce704f8e3947e1f8b2ebb87de9fc4" exitCode=6
Jul 17 13:13:28.936094 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:28.935679    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw" event={"ID":"fbb02592-f3cb-4062-bfbc-f3e3f68c5656","Type":"ContainerDied","Data":"5fe9e2a178cc935ecc755625e9e0a518b13ce704f8e3947e1f8b2ebb87de9fc4"}
Jul 17 13:13:28.936094 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:28.935742    2569 scope.go:117] "RemoveContainer" containerID="19c219c385c6d15f4a8a817a43fcbc9d8a8088858ea992d63d275708a77f140b"
Jul 17 13:13:28.936094 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:28.936044    2569 scope.go:117] "RemoveContainer" containerID="5fe9e2a178cc935ecc755625e9e0a518b13ce704f8e3947e1f8b2ebb87de9fc4"
Jul 17 13:13:28.936278 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:28.936258    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw_odh-ai-gateway-infra(fbb02592-f3cb-4062-bfbc-f3e3f68c5656)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw" podUID="fbb02592-f3cb-4062-bfbc-f3e3f68c5656"
Jul 17 13:13:29.973414 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:29.973379    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw"]
Jul 17 13:13:30.107965 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:30.107941    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw"
Jul 17 13:13:30.160902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:30.160875    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb02592-f3cb-4062-bfbc-f3e3f68c5656-maas-api-service-ca\") pod \"fbb02592-f3cb-4062-bfbc-f3e3f68c5656\" (UID: \"fbb02592-f3cb-4062-bfbc-f3e3f68c5656\") "
Jul 17 13:13:30.161204 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:30.161181    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb02592-f3cb-4062-bfbc-f3e3f68c5656-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fbb02592-f3cb-4062-bfbc-f3e3f68c5656" (UID: "fbb02592-f3cb-4062-bfbc-f3e3f68c5656"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:13:30.262332 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:30.262310    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb02592-f3cb-4062-bfbc-f3e3f68c5656-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:13:30.948121 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:30.948096    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw"
Jul 17 13:13:30.948121 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:30.948109    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw" event={"ID":"fbb02592-f3cb-4062-bfbc-f3e3f68c5656","Type":"ContainerDied","Data":"9e88482539125bdfda83ee6383680b9913cb512af856ed0763eb455de2682721"}
Jul 17 13:13:30.948334 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:30.948165    2569 scope.go:117] "RemoveContainer" containerID="5fe9e2a178cc935ecc755625e9e0a518b13ce704f8e3947e1f8b2ebb87de9fc4"
Jul 17 13:13:30.990533 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:30.990510    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw"]
Jul 17 13:13:31.000122 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.000097    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-njdcw"]
Jul 17 13:13:31.634341 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.634308    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbb02592-f3cb-4062-bfbc-f3e3f68c5656" path="/var/lib/kubelet/pods/fbb02592-f3cb-4062-bfbc-f3e3f68c5656/volumes"
Jul 17 13:13:31.858884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.858855    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r"]
Jul 17 13:13:31.859176 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859163    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:31.859222 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859177    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:31.859222 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859186    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:31.859222 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859192    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:31.859222 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859200    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:31.859222 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859205    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:31.859222 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859210    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbb02592-f3cb-4062-bfbc-f3e3f68c5656" containerName="revoke-keys"
Jul 17 13:13:31.859222 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859215    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbb02592-f3cb-4062-bfbc-f3e3f68c5656" containerName="revoke-keys"
Jul 17 13:13:31.859222 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859221    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:31.859222 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859226    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:31.859486 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859238    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbb02592-f3cb-4062-bfbc-f3e3f68c5656" containerName="revoke-keys"
Jul 17 13:13:31.859486 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859243    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbb02592-f3cb-4062-bfbc-f3e3f68c5656" containerName="revoke-keys"
Jul 17 13:13:31.859486 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859304    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:31.859486 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859311    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:31.859486 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859317    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:31.859486 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859324    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbb02592-f3cb-4062-bfbc-f3e3f68c5656" containerName="revoke-keys"
Jul 17 13:13:31.859486 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859329    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbb02592-f3cb-4062-bfbc-f3e3f68c5656" containerName="revoke-keys"
Jul 17 13:13:31.859486 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859336    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:31.859486 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859341    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="34255402-5b50-4c99-918d-ed3f0711a238" containerName="revoke-keys"
Jul 17 13:13:31.859486 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.859347    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:31.863598 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.863581    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r"
Jul 17 13:13:31.867226 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.867206    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:13:31.867326 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.867264    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:13:31.876744 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.876720    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r"]
Jul 17 13:13:31.976352 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:31.976331    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a232d47-d2f4-4aaf-a137-381abdf82580-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r\" (UID: \"4a232d47-d2f4-4aaf-a137-381abdf82580\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r"
Jul 17 13:13:32.076938 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:32.076912    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a232d47-d2f4-4aaf-a137-381abdf82580-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r\" (UID: \"4a232d47-d2f4-4aaf-a137-381abdf82580\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r"
Jul 17 13:13:32.077431 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:32.077412    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a232d47-d2f4-4aaf-a137-381abdf82580-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r\" (UID: \"4a232d47-d2f4-4aaf-a137-381abdf82580\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r"
Jul 17 13:13:32.173850 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:32.173828    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r"
Jul 17 13:13:32.307190 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:32.307164    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r"]
Jul 17 13:13:32.309625 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:13:32.309599    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a232d47_d2f4_4aaf_a137_381abdf82580.slice/crio-a94e536d8377b9eea5b69fbd3813a6a18a52eed282fbeaf117c5ac6788413471 WatchSource:0}: Error finding container a94e536d8377b9eea5b69fbd3813a6a18a52eed282fbeaf117c5ac6788413471: Status 404 returned error can't find the container with id a94e536d8377b9eea5b69fbd3813a6a18a52eed282fbeaf117c5ac6788413471
Jul 17 13:13:32.959894 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:32.959866    2569 generic.go:356] "Generic (PLEG): container finished" podID="4a232d47-d2f4-4aaf-a137-381abdf82580" containerID="a22b8f65463469c3798665be8ec5a3416d8435737835e4c507e5e31b5514be70" exitCode=6
Jul 17 13:13:32.960011 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:32.959960    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r" event={"ID":"4a232d47-d2f4-4aaf-a137-381abdf82580","Type":"ContainerDied","Data":"a22b8f65463469c3798665be8ec5a3416d8435737835e4c507e5e31b5514be70"}
Jul 17 13:13:32.960080 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:32.960021    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r" event={"ID":"4a232d47-d2f4-4aaf-a137-381abdf82580","Type":"ContainerStarted","Data":"a94e536d8377b9eea5b69fbd3813a6a18a52eed282fbeaf117c5ac6788413471"}
Jul 17 13:13:32.960080 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:32.960050    2569 scope.go:117] "RemoveContainer" containerID="a22b8f65463469c3798665be8ec5a3416d8435737835e4c507e5e31b5514be70"
Jul 17 13:13:33.857230 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.857144    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"]
Jul 17 13:13:33.858066 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.857448    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbb02592-f3cb-4062-bfbc-f3e3f68c5656" containerName="revoke-keys"
Jul 17 13:13:33.858066 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.857458    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbb02592-f3cb-4062-bfbc-f3e3f68c5656" containerName="revoke-keys"
Jul 17 13:13:33.858066 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.857478    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:33.858066 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.857483    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:33.858066 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.857491    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:33.858066 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.857497    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:33.858066 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.857554    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:33.858066 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.857562    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbb02592-f3cb-4062-bfbc-f3e3f68c5656" containerName="revoke-keys"
Jul 17 13:13:33.858066 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.857568    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:33.860440 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.860422    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:13:33.864660 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.864639    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-661ee8a2-serving-cert\""
Jul 17 13:13:33.864827 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.864808    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 13:13:33.865660 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.865639    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lwvzj\""
Jul 17 13:13:33.886644 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.886623    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"]
Jul 17 13:13:33.890963 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.890940    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c1b598d-1c76-4afd-8c32-c50b57a701c5-maas-api-tls\") pod \"maas-api-e2e-derive-661ee8a2-594dbcc585-js747\" (UID: \"2c1b598d-1c76-4afd-8c32-c50b57a701c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:13:33.891078 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.890997    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1b598d-1c76-4afd-8c32-c50b57a701c5-openshift-service-ca\") pod \"maas-api-e2e-derive-661ee8a2-594dbcc585-js747\" (UID: \"2c1b598d-1c76-4afd-8c32-c50b57a701c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:13:33.891151 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.891096    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fd9w\" (UniqueName: \"kubernetes.io/projected/2c1b598d-1c76-4afd-8c32-c50b57a701c5-kube-api-access-9fd9w\") pod \"maas-api-e2e-derive-661ee8a2-594dbcc585-js747\" (UID: \"2c1b598d-1c76-4afd-8c32-c50b57a701c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:13:33.966825 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.966798    2569 generic.go:356] "Generic (PLEG): container finished" podID="4a232d47-d2f4-4aaf-a137-381abdf82580" containerID="24ed67b6939c9d0a8507e120251d6e0d32eccadd9375c4364be929699535c91b" exitCode=6
Jul 17 13:13:33.966927 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.966883    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r" event={"ID":"4a232d47-d2f4-4aaf-a137-381abdf82580","Type":"ContainerDied","Data":"24ed67b6939c9d0a8507e120251d6e0d32eccadd9375c4364be929699535c91b"}
Jul 17 13:13:33.966976 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.966937    2569 scope.go:117] "RemoveContainer" containerID="a22b8f65463469c3798665be8ec5a3416d8435737835e4c507e5e31b5514be70"
Jul 17 13:13:33.967141 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.967126    2569 scope.go:117] "RemoveContainer" containerID="24ed67b6939c9d0a8507e120251d6e0d32eccadd9375c4364be929699535c91b"
Jul 17 13:13:33.967340 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:33.967323    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r_odh-ai-gateway-infra(4a232d47-d2f4-4aaf-a137-381abdf82580)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r" podUID="4a232d47-d2f4-4aaf-a137-381abdf82580"
Jul 17 13:13:33.992047 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.992017    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1b598d-1c76-4afd-8c32-c50b57a701c5-openshift-service-ca\") pod \"maas-api-e2e-derive-661ee8a2-594dbcc585-js747\" (UID: \"2c1b598d-1c76-4afd-8c32-c50b57a701c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:13:33.992167 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.992073    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fd9w\" (UniqueName: \"kubernetes.io/projected/2c1b598d-1c76-4afd-8c32-c50b57a701c5-kube-api-access-9fd9w\") pod \"maas-api-e2e-derive-661ee8a2-594dbcc585-js747\" (UID: \"2c1b598d-1c76-4afd-8c32-c50b57a701c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:13:33.992167 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.992154    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c1b598d-1c76-4afd-8c32-c50b57a701c5-maas-api-tls\") pod \"maas-api-e2e-derive-661ee8a2-594dbcc585-js747\" (UID: \"2c1b598d-1c76-4afd-8c32-c50b57a701c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:13:33.992665 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.992644    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1b598d-1c76-4afd-8c32-c50b57a701c5-openshift-service-ca\") pod \"maas-api-e2e-derive-661ee8a2-594dbcc585-js747\" (UID: \"2c1b598d-1c76-4afd-8c32-c50b57a701c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:13:33.994308 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:33.994287    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c1b598d-1c76-4afd-8c32-c50b57a701c5-maas-api-tls\") pod \"maas-api-e2e-derive-661ee8a2-594dbcc585-js747\" (UID: \"2c1b598d-1c76-4afd-8c32-c50b57a701c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:13:34.011049 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:34.011030    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fd9w\" (UniqueName: \"kubernetes.io/projected/2c1b598d-1c76-4afd-8c32-c50b57a701c5-kube-api-access-9fd9w\") pod \"maas-api-e2e-derive-661ee8a2-594dbcc585-js747\" (UID: \"2c1b598d-1c76-4afd-8c32-c50b57a701c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:13:34.171856 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:34.171796    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:13:34.313637 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:34.313616    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"]
Jul 17 13:13:34.316442 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:13:34.316416    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1b598d_1c76_4afd_8c32_c50b57a701c5.slice/crio-df33c8d58ce20864f5a03b6bc4ea168e3145547487126c80fb80f5cf37ef01d8 WatchSource:0}: Error finding container df33c8d58ce20864f5a03b6bc4ea168e3145547487126c80fb80f5cf37ef01d8: Status 404 returned error can't find the container with id df33c8d58ce20864f5a03b6bc4ea168e3145547487126c80fb80f5cf37ef01d8
Jul 17 13:13:34.973013 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:34.972945    2569 scope.go:117] "RemoveContainer" containerID="24ed67b6939c9d0a8507e120251d6e0d32eccadd9375c4364be929699535c91b"
Jul 17 13:13:34.973385 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:34.973130    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r_odh-ai-gateway-infra(4a232d47-d2f4-4aaf-a137-381abdf82580)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r" podUID="4a232d47-d2f4-4aaf-a137-381abdf82580"
Jul 17 13:13:34.974305 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:34.974282    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" event={"ID":"2c1b598d-1c76-4afd-8c32-c50b57a701c5","Type":"ContainerStarted","Data":"331c763e42fffceec6b644ccd8b64f2fd3771b4c9938d498be99648903d6a17a"}
Jul 17 13:13:34.974414 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:34.974325    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" event={"ID":"2c1b598d-1c76-4afd-8c32-c50b57a701c5","Type":"ContainerStarted","Data":"df33c8d58ce20864f5a03b6bc4ea168e3145547487126c80fb80f5cf37ef01d8"}
Jul 17 13:13:34.974414 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:34.974344    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:13:35.014009 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:35.013958    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podStartSLOduration=1.622231634 podStartE2EDuration="2.013944635s" totalImagesPullingTime="391.713001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:13:35.01235962 +0000 UTC m=+3090.016406502" watchObservedRunningTime="2026-07-17 13:13:35.013944635 +0000 UTC m=+3090.017991516"
Jul 17 13:13:35.980212 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:35.980177    2569 generic.go:356] "Generic (PLEG): container finished" podID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerID="331c763e42fffceec6b644ccd8b64f2fd3771b4c9938d498be99648903d6a17a" exitCode=1
Jul 17 13:13:35.980569 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:35.980263    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" event={"ID":"2c1b598d-1c76-4afd-8c32-c50b57a701c5","Type":"ContainerDied","Data":"331c763e42fffceec6b644ccd8b64f2fd3771b4c9938d498be99648903d6a17a"}
Jul 17 13:13:35.980569 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:35.980413    2569 scope.go:117] "RemoveContainer" containerID="331c763e42fffceec6b644ccd8b64f2fd3771b4c9938d498be99648903d6a17a"
Jul 17 13:13:36.265317 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:36.265224    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:13:36.986941 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:36.986912    2569 generic.go:356] "Generic (PLEG): container finished" podID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerID="13bce4def8ebbd8a3830f3a3e053889d1474e51575fc3f5fca36cfa8baf506fb" exitCode=1
Jul 17 13:13:36.987319 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:36.986998    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" event={"ID":"2c1b598d-1c76-4afd-8c32-c50b57a701c5","Type":"ContainerDied","Data":"13bce4def8ebbd8a3830f3a3e053889d1474e51575fc3f5fca36cfa8baf506fb"}
Jul 17 13:13:36.987319 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:36.987058    2569 scope.go:117] "RemoveContainer" containerID="331c763e42fffceec6b644ccd8b64f2fd3771b4c9938d498be99648903d6a17a"
Jul 17 13:13:36.987319 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:36.987235    2569 scope.go:117] "RemoveContainer" containerID="13bce4def8ebbd8a3830f3a3e053889d1474e51575fc3f5fca36cfa8baf506fb"
Jul 17 13:13:36.987479 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:36.987444    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:13:37.994551 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:37.994524    2569 scope.go:117] "RemoveContainer" containerID="13bce4def8ebbd8a3830f3a3e053889d1474e51575fc3f5fca36cfa8baf506fb"
Jul 17 13:13:37.994977 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:37.994685    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:13:38.190057 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:38.190028    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228"]
Jul 17 13:13:38.190391 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:38.190375    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:38.190430 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:38.190393    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcff1798-e8d8-4449-b8e3-eaf723b636c4" containerName="maas-api"
Jul 17 13:13:38.194502 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:38.194486    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228"
Jul 17 13:13:38.203845 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:38.203820    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228"]
Jul 17 13:13:38.228000 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:38.227971    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c3e4d7-27ca-4093-8118-04e00173b2f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-ng228\" (UID: \"55c3e4d7-27ca-4093-8118-04e00173b2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228"
Jul 17 13:13:38.328301 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:38.328243    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c3e4d7-27ca-4093-8118-04e00173b2f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-ng228\" (UID: \"55c3e4d7-27ca-4093-8118-04e00173b2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228"
Jul 17 13:13:38.328812 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:38.328795    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c3e4d7-27ca-4093-8118-04e00173b2f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-ng228\" (UID: \"55c3e4d7-27ca-4093-8118-04e00173b2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228"
Jul 17 13:13:38.505910 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:38.505884    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228"
Jul 17 13:13:38.843894 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:38.843870    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228"]
Jul 17 13:13:38.846090 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:13:38.846063    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c3e4d7_27ca_4093_8118_04e00173b2f2.slice/crio-9a11f09f7ce35b363321218862585bbafbb8ce70d1988a6f61d7d1e2613cf79c WatchSource:0}: Error finding container 9a11f09f7ce35b363321218862585bbafbb8ce70d1988a6f61d7d1e2613cf79c: Status 404 returned error can't find the container with id 9a11f09f7ce35b363321218862585bbafbb8ce70d1988a6f61d7d1e2613cf79c
Jul 17 13:13:39.000275 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:39.000244    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228" event={"ID":"55c3e4d7-27ca-4093-8118-04e00173b2f2","Type":"ContainerStarted","Data":"9a11f09f7ce35b363321218862585bbafbb8ce70d1988a6f61d7d1e2613cf79c"}
Jul 17 13:13:39.974663 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:39.974631    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:13:39.975080 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:39.975063    2569 scope.go:117] "RemoveContainer" containerID="13bce4def8ebbd8a3830f3a3e053889d1474e51575fc3f5fca36cfa8baf506fb"
Jul 17 13:13:39.975257 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:39.975238    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:13:40.006504 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:40.006468    2569 generic.go:356] "Generic (PLEG): container finished" podID="55c3e4d7-27ca-4093-8118-04e00173b2f2" containerID="3fb6e10a45bbad640faec89ca614e6f6450e25e9fc69eef522fe1a2cd860ea44" exitCode=7
Jul 17 13:13:40.006890 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:40.006542    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228" event={"ID":"55c3e4d7-27ca-4093-8118-04e00173b2f2","Type":"ContainerDied","Data":"3fb6e10a45bbad640faec89ca614e6f6450e25e9fc69eef522fe1a2cd860ea44"}
Jul 17 13:13:40.006890 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:40.006695    2569 scope.go:117] "RemoveContainer" containerID="3fb6e10a45bbad640faec89ca614e6f6450e25e9fc69eef522fe1a2cd860ea44"
Jul 17 13:13:40.691950 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:40.691922    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj"]
Jul 17 13:13:40.695720 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:40.695701    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj"
Jul 17 13:13:40.703424 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:40.703366    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj"]
Jul 17 13:13:40.749072 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:40.749039    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00da379e-2080-4bd7-8ef3-ddf1bb7b2a74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj\" (UID: \"00da379e-2080-4bd7-8ef3-ddf1bb7b2a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj"
Jul 17 13:13:40.850020 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:40.849954    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00da379e-2080-4bd7-8ef3-ddf1bb7b2a74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj\" (UID: \"00da379e-2080-4bd7-8ef3-ddf1bb7b2a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj"
Jul 17 13:13:40.850520 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:40.850497    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00da379e-2080-4bd7-8ef3-ddf1bb7b2a74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj\" (UID: \"00da379e-2080-4bd7-8ef3-ddf1bb7b2a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj"
Jul 17 13:13:41.007512 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:41.007480    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj"
Jul 17 13:13:41.013620 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:41.013587    2569 generic.go:356] "Generic (PLEG): container finished" podID="55c3e4d7-27ca-4093-8118-04e00173b2f2" containerID="3c52ede82401225d108a4b45285bb1c50e018f624df8dce467a9a52ed930c7dc" exitCode=7
Jul 17 13:13:41.013728 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:41.013641    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228" event={"ID":"55c3e4d7-27ca-4093-8118-04e00173b2f2","Type":"ContainerDied","Data":"3c52ede82401225d108a4b45285bb1c50e018f624df8dce467a9a52ed930c7dc"}
Jul 17 13:13:41.013728 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:41.013695    2569 scope.go:117] "RemoveContainer" containerID="3fb6e10a45bbad640faec89ca614e6f6450e25e9fc69eef522fe1a2cd860ea44"
Jul 17 13:13:41.013899 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:41.013880    2569 scope.go:117] "RemoveContainer" containerID="3c52ede82401225d108a4b45285bb1c50e018f624df8dce467a9a52ed930c7dc"
Jul 17 13:13:41.014104 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:41.014080    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-ng228_odh-ai-gateway-infra(55c3e4d7-27ca-4093-8118-04e00173b2f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228" podUID="55c3e4d7-27ca-4093-8118-04e00173b2f2"
Jul 17 13:13:41.136550 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:41.136481    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj"]
Jul 17 13:13:41.139645 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:13:41.139612    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00da379e_2080_4bd7_8ef3_ddf1bb7b2a74.slice/crio-dc75bc6997d629866be3f25aa4aa79a37cc44460f4da5543c12ba2440664c06f WatchSource:0}: Error finding container dc75bc6997d629866be3f25aa4aa79a37cc44460f4da5543c12ba2440664c06f: Status 404 returned error can't find the container with id dc75bc6997d629866be3f25aa4aa79a37cc44460f4da5543c12ba2440664c06f
Jul 17 13:13:42.019730 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:42.019663    2569 scope.go:117] "RemoveContainer" containerID="3c52ede82401225d108a4b45285bb1c50e018f624df8dce467a9a52ed930c7dc"
Jul 17 13:13:42.020105 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:42.019883    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-ng228_odh-ai-gateway-infra(55c3e4d7-27ca-4093-8118-04e00173b2f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228" podUID="55c3e4d7-27ca-4093-8118-04e00173b2f2"
Jul 17 13:13:42.021073 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:42.021051    2569 generic.go:356] "Generic (PLEG): container finished" podID="00da379e-2080-4bd7-8ef3-ddf1bb7b2a74" containerID="65f48c28c0acc3e73b23cffd59bfb737a288806e28a0cb3391c6c4d2fa947154" exitCode=6
Jul 17 13:13:42.021158 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:42.021138    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj" event={"ID":"00da379e-2080-4bd7-8ef3-ddf1bb7b2a74","Type":"ContainerDied","Data":"65f48c28c0acc3e73b23cffd59bfb737a288806e28a0cb3391c6c4d2fa947154"}
Jul 17 13:13:42.021205 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:42.021191    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj" event={"ID":"00da379e-2080-4bd7-8ef3-ddf1bb7b2a74","Type":"ContainerStarted","Data":"dc75bc6997d629866be3f25aa4aa79a37cc44460f4da5543c12ba2440664c06f"}
Jul 17 13:13:42.021271 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:42.021257    2569 scope.go:117] "RemoveContainer" containerID="65f48c28c0acc3e73b23cffd59bfb737a288806e28a0cb3391c6c4d2fa947154"
Jul 17 13:13:43.029369 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:43.029337    2569 generic.go:356] "Generic (PLEG): container finished" podID="00da379e-2080-4bd7-8ef3-ddf1bb7b2a74" containerID="d4470c6b3deb600aa7e67661819824ba1b83337e5b5c505693a25131f6055c4e" exitCode=6
Jul 17 13:13:43.029710 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:43.029425    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj" event={"ID":"00da379e-2080-4bd7-8ef3-ddf1bb7b2a74","Type":"ContainerDied","Data":"d4470c6b3deb600aa7e67661819824ba1b83337e5b5c505693a25131f6055c4e"}
Jul 17 13:13:43.029710 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:43.029494    2569 scope.go:117] "RemoveContainer" containerID="65f48c28c0acc3e73b23cffd59bfb737a288806e28a0cb3391c6c4d2fa947154"
Jul 17 13:13:43.029710 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:43.029594    2569 scope.go:117] "RemoveContainer" containerID="d4470c6b3deb600aa7e67661819824ba1b83337e5b5c505693a25131f6055c4e"
Jul 17 13:13:43.029889 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:43.029799    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj_odh-ai-gateway-infra(00da379e-2080-4bd7-8ef3-ddf1bb7b2a74)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj" podUID="00da379e-2080-4bd7-8ef3-ddf1bb7b2a74"
Jul 17 13:13:44.035159 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:44.035130    2569 scope.go:117] "RemoveContainer" containerID="d4470c6b3deb600aa7e67661819824ba1b83337e5b5c505693a25131f6055c4e"
Jul 17 13:13:44.035547 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:44.035299    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj_odh-ai-gateway-infra(00da379e-2080-4bd7-8ef3-ddf1bb7b2a74)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj" podUID="00da379e-2080-4bd7-8ef3-ddf1bb7b2a74"
Jul 17 13:13:44.172438 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:44.172411    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:13:44.172734 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:44.172721    2569 scope.go:117] "RemoveContainer" containerID="13bce4def8ebbd8a3830f3a3e053889d1474e51575fc3f5fca36cfa8baf506fb"
Jul 17 13:13:44.172928 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:44.172911    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:13:49.629378 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:49.629346    2569 scope.go:117] "RemoveContainer" containerID="24ed67b6939c9d0a8507e120251d6e0d32eccadd9375c4364be929699535c91b"
Jul 17 13:13:50.064957 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:50.064923    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r" event={"ID":"4a232d47-d2f4-4aaf-a137-381abdf82580","Type":"ContainerStarted","Data":"630884e3b066722a71ed0c5ce942cdc73aa327a5135d59c529b1e149f4011830"}
Jul 17 13:13:50.094963 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:50.094917    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r" podStartSLOduration=19.094904945 podStartE2EDuration="19.094904945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:13:50.093910933 +0000 UTC m=+3105.097957816" watchObservedRunningTime="2026-07-17 13:13:50.094904945 +0000 UTC m=+3105.098951805"
Jul 17 13:13:51.071991 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:51.071958    2569 generic.go:356] "Generic (PLEG): container finished" podID="4a232d47-d2f4-4aaf-a137-381abdf82580" containerID="630884e3b066722a71ed0c5ce942cdc73aa327a5135d59c529b1e149f4011830" exitCode=6
Jul 17 13:13:51.072382 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:51.072027    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r" event={"ID":"4a232d47-d2f4-4aaf-a137-381abdf82580","Type":"ContainerDied","Data":"630884e3b066722a71ed0c5ce942cdc73aa327a5135d59c529b1e149f4011830"}
Jul 17 13:13:51.072382 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:51.072096    2569 scope.go:117] "RemoveContainer" containerID="24ed67b6939c9d0a8507e120251d6e0d32eccadd9375c4364be929699535c91b"
Jul 17 13:13:51.072382 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:51.072327    2569 scope.go:117] "RemoveContainer" containerID="630884e3b066722a71ed0c5ce942cdc73aa327a5135d59c529b1e149f4011830"
Jul 17 13:13:51.072557 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:51.072531    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r_odh-ai-gateway-infra(4a232d47-d2f4-4aaf-a137-381abdf82580)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r" podUID="4a232d47-d2f4-4aaf-a137-381abdf82580"
Jul 17 13:13:51.104211 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:51.104186    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r"]
Jul 17 13:13:52.212977 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:52.212954    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r"
Jul 17 13:13:52.337863 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:52.337810    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a232d47-d2f4-4aaf-a137-381abdf82580-maas-api-service-ca\") pod \"4a232d47-d2f4-4aaf-a137-381abdf82580\" (UID: \"4a232d47-d2f4-4aaf-a137-381abdf82580\") "
Jul 17 13:13:52.338166 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:52.338142    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a232d47-d2f4-4aaf-a137-381abdf82580-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4a232d47-d2f4-4aaf-a137-381abdf82580" (UID: "4a232d47-d2f4-4aaf-a137-381abdf82580"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:13:52.438875 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:52.438853    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a232d47-d2f4-4aaf-a137-381abdf82580-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:13:53.084582 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:53.084554    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r"
Jul 17 13:13:53.084708 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:53.084594    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r" event={"ID":"4a232d47-d2f4-4aaf-a137-381abdf82580","Type":"ContainerDied","Data":"a94e536d8377b9eea5b69fbd3813a6a18a52eed282fbeaf117c5ac6788413471"}
Jul 17 13:13:53.084708 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:53.084670    2569 scope.go:117] "RemoveContainer" containerID="630884e3b066722a71ed0c5ce942cdc73aa327a5135d59c529b1e149f4011830"
Jul 17 13:13:53.108941 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:53.108917    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r"]
Jul 17 13:13:53.112578 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:53.112550    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-jsb7r"]
Jul 17 13:13:53.634030 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:53.634005    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a232d47-d2f4-4aaf-a137-381abdf82580" path="/var/lib/kubelet/pods/4a232d47-d2f4-4aaf-a137-381abdf82580/volumes"
Jul 17 13:13:54.628713 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:54.628681    2569 scope.go:117] "RemoveContainer" containerID="3c52ede82401225d108a4b45285bb1c50e018f624df8dce467a9a52ed930c7dc"
Jul 17 13:13:54.628914 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:54.628816    2569 scope.go:117] "RemoveContainer" containerID="d4470c6b3deb600aa7e67661819824ba1b83337e5b5c505693a25131f6055c4e"
Jul 17 13:13:55.095866 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:55.095836    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj" event={"ID":"00da379e-2080-4bd7-8ef3-ddf1bb7b2a74","Type":"ContainerStarted","Data":"ddd781884994eb083ca8ac0356a9be2854336d65bb78cfe830117976a5e6c8c7"}
Jul 17 13:13:55.097722 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:55.097702    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228" event={"ID":"55c3e4d7-27ca-4093-8118-04e00173b2f2","Type":"ContainerStarted","Data":"6e5d12bd1aff0b9c65aebd3f56585a38770f10b42ed9a3ba7d0a445557781fa3"}
Jul 17 13:13:55.111519 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:55.111475    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj" podStartSLOduration=15.11146226 podStartE2EDuration="15.11146226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:13:55.10965957 +0000 UTC m=+3110.113706451" watchObservedRunningTime="2026-07-17 13:13:55.11146226 +0000 UTC m=+3110.115509142"
Jul 17 13:13:55.126769 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:55.126503    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228" podStartSLOduration=17.126485384 podStartE2EDuration="17.126485384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:13:55.12323445 +0000 UTC m=+3110.127281326" watchObservedRunningTime="2026-07-17 13:13:55.126485384 +0000 UTC m=+3110.130532268"
Jul 17 13:13:56.104793 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:56.104731    2569 generic.go:356] "Generic (PLEG): container finished" podID="55c3e4d7-27ca-4093-8118-04e00173b2f2" containerID="6e5d12bd1aff0b9c65aebd3f56585a38770f10b42ed9a3ba7d0a445557781fa3" exitCode=7
Jul 17 13:13:56.105270 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:56.104805    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228" event={"ID":"55c3e4d7-27ca-4093-8118-04e00173b2f2","Type":"ContainerDied","Data":"6e5d12bd1aff0b9c65aebd3f56585a38770f10b42ed9a3ba7d0a445557781fa3"}
Jul 17 13:13:56.105270 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:56.104879    2569 scope.go:117] "RemoveContainer" containerID="3c52ede82401225d108a4b45285bb1c50e018f624df8dce467a9a52ed930c7dc"
Jul 17 13:13:56.105270 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:56.105139    2569 scope.go:117] "RemoveContainer" containerID="6e5d12bd1aff0b9c65aebd3f56585a38770f10b42ed9a3ba7d0a445557781fa3"
Jul 17 13:13:56.105420 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:56.105340    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-ng228_odh-ai-gateway-infra(55c3e4d7-27ca-4093-8118-04e00173b2f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228" podUID="55c3e4d7-27ca-4093-8118-04e00173b2f2"
Jul 17 13:13:56.107011 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:56.106984    2569 generic.go:356] "Generic (PLEG): container finished" podID="00da379e-2080-4bd7-8ef3-ddf1bb7b2a74" containerID="ddd781884994eb083ca8ac0356a9be2854336d65bb78cfe830117976a5e6c8c7" exitCode=6
Jul 17 13:13:56.107118 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:56.107028    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj" event={"ID":"00da379e-2080-4bd7-8ef3-ddf1bb7b2a74","Type":"ContainerDied","Data":"ddd781884994eb083ca8ac0356a9be2854336d65bb78cfe830117976a5e6c8c7"}
Jul 17 13:13:56.107285 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:56.107271    2569 scope.go:117] "RemoveContainer" containerID="ddd781884994eb083ca8ac0356a9be2854336d65bb78cfe830117976a5e6c8c7"
Jul 17 13:13:56.107456 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:56.107439    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj_odh-ai-gateway-infra(00da379e-2080-4bd7-8ef3-ddf1bb7b2a74)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj" podUID="00da379e-2080-4bd7-8ef3-ddf1bb7b2a74"
Jul 17 13:13:56.116530 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:56.116511    2569 scope.go:117] "RemoveContainer" containerID="d4470c6b3deb600aa7e67661819824ba1b83337e5b5c505693a25131f6055c4e"
Jul 17 13:13:56.119839 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:56.119805    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj"]
Jul 17 13:13:56.132203 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:56.132177    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228"]
Jul 17 13:13:56.628163 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:56.628126    2569 scope.go:117] "RemoveContainer" containerID="13bce4def8ebbd8a3830f3a3e053889d1474e51575fc3f5fca36cfa8baf506fb"
Jul 17 13:13:57.112849 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:57.112818    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" event={"ID":"2c1b598d-1c76-4afd-8c32-c50b57a701c5","Type":"ContainerStarted","Data":"40e3f3ef3703945879ff831a0d691d896a7fde7ab33dc3f2b7517d48a86f7fa7"}
Jul 17 13:13:57.113221 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:57.113017    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:13:57.256419 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:57.256389    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj"
Jul 17 13:13:57.282374 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:57.282353    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228"
Jul 17 13:13:57.376631 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:57.376602    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00da379e-2080-4bd7-8ef3-ddf1bb7b2a74-maas-api-service-ca\") pod \"00da379e-2080-4bd7-8ef3-ddf1bb7b2a74\" (UID: \"00da379e-2080-4bd7-8ef3-ddf1bb7b2a74\") "
Jul 17 13:13:57.376732 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:57.376690    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c3e4d7-27ca-4093-8118-04e00173b2f2-maas-api-service-ca\") pod \"55c3e4d7-27ca-4093-8118-04e00173b2f2\" (UID: \"55c3e4d7-27ca-4093-8118-04e00173b2f2\") "
Jul 17 13:13:57.376978 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:57.376957    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00da379e-2080-4bd7-8ef3-ddf1bb7b2a74-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00da379e-2080-4bd7-8ef3-ddf1bb7b2a74" (UID: "00da379e-2080-4bd7-8ef3-ddf1bb7b2a74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:13:57.377024 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:57.377005    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c3e4d7-27ca-4093-8118-04e00173b2f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55c3e4d7-27ca-4093-8118-04e00173b2f2" (UID: "55c3e4d7-27ca-4093-8118-04e00173b2f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:13:57.477216 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:57.477192    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55c3e4d7-27ca-4093-8118-04e00173b2f2-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:13:57.477216 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:57.477213    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00da379e-2080-4bd7-8ef3-ddf1bb7b2a74-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:13:58.122077 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:58.122041    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228"
Jul 17 13:13:58.122077 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:58.122052    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228" event={"ID":"55c3e4d7-27ca-4093-8118-04e00173b2f2","Type":"ContainerDied","Data":"9a11f09f7ce35b363321218862585bbafbb8ce70d1988a6f61d7d1e2613cf79c"}
Jul 17 13:13:58.122541 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:58.122126    2569 scope.go:117] "RemoveContainer" containerID="6e5d12bd1aff0b9c65aebd3f56585a38770f10b42ed9a3ba7d0a445557781fa3"
Jul 17 13:13:58.123699 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:58.123673    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj"
Jul 17 13:13:58.123699 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:58.123686    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj" event={"ID":"00da379e-2080-4bd7-8ef3-ddf1bb7b2a74","Type":"ContainerDied","Data":"dc75bc6997d629866be3f25aa4aa79a37cc44460f4da5543c12ba2440664c06f"}
Jul 17 13:13:58.126454 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:58.126425    2569 generic.go:356] "Generic (PLEG): container finished" podID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerID="40e3f3ef3703945879ff831a0d691d896a7fde7ab33dc3f2b7517d48a86f7fa7" exitCode=1
Jul 17 13:13:58.126581 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:58.126457    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" event={"ID":"2c1b598d-1c76-4afd-8c32-c50b57a701c5","Type":"ContainerDied","Data":"40e3f3ef3703945879ff831a0d691d896a7fde7ab33dc3f2b7517d48a86f7fa7"}
Jul 17 13:13:58.126733 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:58.126714    2569 scope.go:117] "RemoveContainer" containerID="40e3f3ef3703945879ff831a0d691d896a7fde7ab33dc3f2b7517d48a86f7fa7"
Jul 17 13:13:58.126936 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:58.126917    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:13:58.131878 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:58.131857    2569 scope.go:117] "RemoveContainer" containerID="ddd781884994eb083ca8ac0356a9be2854336d65bb78cfe830117976a5e6c8c7"
Jul 17 13:13:58.139835 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:58.139817    2569 scope.go:117] "RemoveContainer" containerID="13bce4def8ebbd8a3830f3a3e053889d1474e51575fc3f5fca36cfa8baf506fb"
Jul 17 13:13:58.165867 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:58.165843    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228"]
Jul 17 13:13:58.173012 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:58.172986    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ng228"]
Jul 17 13:13:58.191524 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:58.191501    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj"]
Jul 17 13:13:58.197899 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:58.197879    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k57zj"]
Jul 17 13:13:59.134484 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:59.134461    2569 scope.go:117] "RemoveContainer" containerID="40e3f3ef3703945879ff831a0d691d896a7fde7ab33dc3f2b7517d48a86f7fa7"
Jul 17 13:13:59.134881 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:13:59.134625    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:13:59.634201 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:59.634162    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00da379e-2080-4bd7-8ef3-ddf1bb7b2a74" path="/var/lib/kubelet/pods/00da379e-2080-4bd7-8ef3-ddf1bb7b2a74/volumes"
Jul 17 13:13:59.634546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:13:59.634530    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55c3e4d7-27ca-4093-8118-04e00173b2f2" path="/var/lib/kubelet/pods/55c3e4d7-27ca-4093-8118-04e00173b2f2/volumes"
Jul 17 13:14:04.172303 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:04.172270    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:14:04.172683 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:04.172644    2569 scope.go:117] "RemoveContainer" containerID="40e3f3ef3703945879ff831a0d691d896a7fde7ab33dc3f2b7517d48a86f7fa7"
Jul 17 13:14:04.172869 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:14:04.172848    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:14:18.628880 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:18.628839    2569 scope.go:117] "RemoveContainer" containerID="40e3f3ef3703945879ff831a0d691d896a7fde7ab33dc3f2b7517d48a86f7fa7"
Jul 17 13:14:19.231946 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:19.231914    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" event={"ID":"2c1b598d-1c76-4afd-8c32-c50b57a701c5","Type":"ContainerStarted","Data":"7df4453a14088c4e66a4ac43e075b38ae22ef62802442a5957693e34e346982c"}
Jul 17 13:14:19.232207 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:19.232184    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:14:20.238069 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:20.238033    2569 generic.go:356] "Generic (PLEG): container finished" podID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerID="7df4453a14088c4e66a4ac43e075b38ae22ef62802442a5957693e34e346982c" exitCode=1
Jul 17 13:14:20.238539 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:20.238115    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" event={"ID":"2c1b598d-1c76-4afd-8c32-c50b57a701c5","Type":"ContainerDied","Data":"7df4453a14088c4e66a4ac43e075b38ae22ef62802442a5957693e34e346982c"}
Jul 17 13:14:20.238539 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:20.238188    2569 scope.go:117] "RemoveContainer" containerID="40e3f3ef3703945879ff831a0d691d896a7fde7ab33dc3f2b7517d48a86f7fa7"
Jul 17 13:14:20.238539 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:20.238370    2569 scope.go:117] "RemoveContainer" containerID="7df4453a14088c4e66a4ac43e075b38ae22ef62802442a5957693e34e346982c"
Jul 17 13:14:20.238693 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:14:20.238585    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:14:21.246572 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:21.246538    2569 scope.go:117] "RemoveContainer" containerID="7df4453a14088c4e66a4ac43e075b38ae22ef62802442a5957693e34e346982c"
Jul 17 13:14:21.247028 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:14:21.246725    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:14:24.172464 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:24.172414    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:14:24.172964 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:24.172814    2569 scope.go:117] "RemoveContainer" containerID="7df4453a14088c4e66a4ac43e075b38ae22ef62802442a5957693e34e346982c"
Jul 17 13:14:24.173018 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:14:24.172998    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:14:30.760609 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.760534    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc"]
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.760893    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00da379e-2080-4bd7-8ef3-ddf1bb7b2a74" containerName="revoke-keys"
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.760906    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="00da379e-2080-4bd7-8ef3-ddf1bb7b2a74" containerName="revoke-keys"
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.760914    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c3e4d7-27ca-4093-8118-04e00173b2f2" containerName="revoke-keys"
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.760920    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c3e4d7-27ca-4093-8118-04e00173b2f2" containerName="revoke-keys"
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.760927    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00da379e-2080-4bd7-8ef3-ddf1bb7b2a74" containerName="revoke-keys"
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.760932    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="00da379e-2080-4bd7-8ef3-ddf1bb7b2a74" containerName="revoke-keys"
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.760937    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a232d47-d2f4-4aaf-a137-381abdf82580" containerName="revoke-keys"
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.760942    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a232d47-d2f4-4aaf-a137-381abdf82580" containerName="revoke-keys"
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.760948    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c3e4d7-27ca-4093-8118-04e00173b2f2" containerName="revoke-keys"
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.760953    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c3e4d7-27ca-4093-8118-04e00173b2f2" containerName="revoke-keys"
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.760965    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a232d47-d2f4-4aaf-a137-381abdf82580" containerName="revoke-keys"
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.760971    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a232d47-d2f4-4aaf-a137-381abdf82580" containerName="revoke-keys"
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.761022    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a232d47-d2f4-4aaf-a137-381abdf82580" containerName="revoke-keys"
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.761029    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c3e4d7-27ca-4093-8118-04e00173b2f2" containerName="revoke-keys"
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.761036    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c3e4d7-27ca-4093-8118-04e00173b2f2" containerName="revoke-keys"
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.761043    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="55c3e4d7-27ca-4093-8118-04e00173b2f2" containerName="revoke-keys"
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.761049    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="00da379e-2080-4bd7-8ef3-ddf1bb7b2a74" containerName="revoke-keys"
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.761054    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a232d47-d2f4-4aaf-a137-381abdf82580" containerName="revoke-keys"
Jul 17 13:14:30.761071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.761062    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="00da379e-2080-4bd7-8ef3-ddf1bb7b2a74" containerName="revoke-keys"
Jul 17 13:14:30.765536 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.765515    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc"
Jul 17 13:14:30.768714 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.768691    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:14:30.779534 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.779512    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc"]
Jul 17 13:14:30.832052 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.832032    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/282b8ba5-c7e4-4e83-a683-4e96c1dec6fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc\" (UID: \"282b8ba5-c7e4-4e83-a683-4e96c1dec6fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc"
Jul 17 13:14:30.932371 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.932346    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/282b8ba5-c7e4-4e83-a683-4e96c1dec6fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc\" (UID: \"282b8ba5-c7e4-4e83-a683-4e96c1dec6fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc"
Jul 17 13:14:30.932923 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:30.932905    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/282b8ba5-c7e4-4e83-a683-4e96c1dec6fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc\" (UID: \"282b8ba5-c7e4-4e83-a683-4e96c1dec6fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc"
Jul 17 13:14:31.075998 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:31.075944    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc"
Jul 17 13:14:31.234638 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:31.234612    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc"]
Jul 17 13:14:31.236677 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:14:31.236648    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282b8ba5_c7e4_4e83_a683_4e96c1dec6fd.slice/crio-4cedfa7504abb459a65bc1cef8467d0ac8a4f3cf8f7cfde193ab8daf043415d3 WatchSource:0}: Error finding container 4cedfa7504abb459a65bc1cef8467d0ac8a4f3cf8f7cfde193ab8daf043415d3: Status 404 returned error can't find the container with id 4cedfa7504abb459a65bc1cef8467d0ac8a4f3cf8f7cfde193ab8daf043415d3
Jul 17 13:14:31.292195 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:31.292170    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc" event={"ID":"282b8ba5-c7e4-4e83-a683-4e96c1dec6fd","Type":"ContainerStarted","Data":"4cedfa7504abb459a65bc1cef8467d0ac8a4f3cf8f7cfde193ab8daf043415d3"}
Jul 17 13:14:31.904433 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:31.904402    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j"]
Jul 17 13:14:31.904791 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:31.904776    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00da379e-2080-4bd7-8ef3-ddf1bb7b2a74" containerName="revoke-keys"
Jul 17 13:14:31.904791 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:31.904792    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="00da379e-2080-4bd7-8ef3-ddf1bb7b2a74" containerName="revoke-keys"
Jul 17 13:14:31.904889 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:31.904801    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a232d47-d2f4-4aaf-a137-381abdf82580" containerName="revoke-keys"
Jul 17 13:14:31.904889 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:31.904806    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a232d47-d2f4-4aaf-a137-381abdf82580" containerName="revoke-keys"
Jul 17 13:14:31.904889 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:31.904813    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55c3e4d7-27ca-4093-8118-04e00173b2f2" containerName="revoke-keys"
Jul 17 13:14:31.904889 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:31.904821    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="55c3e4d7-27ca-4093-8118-04e00173b2f2" containerName="revoke-keys"
Jul 17 13:14:31.905017 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:31.904919    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="00da379e-2080-4bd7-8ef3-ddf1bb7b2a74" containerName="revoke-keys"
Jul 17 13:14:31.905017 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:31.904935    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a232d47-d2f4-4aaf-a137-381abdf82580" containerName="revoke-keys"
Jul 17 13:14:31.907855 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:31.907835    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j"
Jul 17 13:14:31.916658 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:31.916616    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j"]
Jul 17 13:14:31.942568 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:31.942544    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b459ab-0b13-4d1d-aabd-03c031865ea5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j\" (UID: \"97b459ab-0b13-4d1d-aabd-03c031865ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j"
Jul 17 13:14:32.043041 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:32.043018    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b459ab-0b13-4d1d-aabd-03c031865ea5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j\" (UID: \"97b459ab-0b13-4d1d-aabd-03c031865ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j"
Jul 17 13:14:32.043588 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:32.043567    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b459ab-0b13-4d1d-aabd-03c031865ea5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j\" (UID: \"97b459ab-0b13-4d1d-aabd-03c031865ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j"
Jul 17 13:14:32.219693 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:32.219624    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j"
Jul 17 13:14:32.298359 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:32.298335    2569 generic.go:356] "Generic (PLEG): container finished" podID="282b8ba5-c7e4-4e83-a683-4e96c1dec6fd" containerID="3de6aa828c502bd77a2834164f28ad9e47ceef628bd0e5403e9d2d54083f880d" exitCode=6
Jul 17 13:14:32.298489 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:32.298421    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc" event={"ID":"282b8ba5-c7e4-4e83-a683-4e96c1dec6fd","Type":"ContainerDied","Data":"3de6aa828c502bd77a2834164f28ad9e47ceef628bd0e5403e9d2d54083f880d"}
Jul 17 13:14:32.298596 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:32.298580    2569 scope.go:117] "RemoveContainer" containerID="3de6aa828c502bd77a2834164f28ad9e47ceef628bd0e5403e9d2d54083f880d"
Jul 17 13:14:32.359508 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:32.359488    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j"]
Jul 17 13:14:32.361006 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:14:32.360983    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b459ab_0b13_4d1d_aabd_03c031865ea5.slice/crio-406eadec55529ccdbb39aef3704b12193f32b3d206f94cf9079fce2b73ac740e WatchSource:0}: Error finding container 406eadec55529ccdbb39aef3704b12193f32b3d206f94cf9079fce2b73ac740e: Status 404 returned error can't find the container with id 406eadec55529ccdbb39aef3704b12193f32b3d206f94cf9079fce2b73ac740e
Jul 17 13:14:33.304923 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:33.304896    2569 generic.go:356] "Generic (PLEG): container finished" podID="282b8ba5-c7e4-4e83-a683-4e96c1dec6fd" containerID="57f1c799303cdb1e42473577202e1ebdb5971a44669225a24260c390480bc850" exitCode=6
Jul 17 13:14:33.305302 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:33.304987    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc" event={"ID":"282b8ba5-c7e4-4e83-a683-4e96c1dec6fd","Type":"ContainerDied","Data":"57f1c799303cdb1e42473577202e1ebdb5971a44669225a24260c390480bc850"}
Jul 17 13:14:33.305302 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:33.305063    2569 scope.go:117] "RemoveContainer" containerID="3de6aa828c502bd77a2834164f28ad9e47ceef628bd0e5403e9d2d54083f880d"
Jul 17 13:14:33.305302 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:33.305205    2569 scope.go:117] "RemoveContainer" containerID="57f1c799303cdb1e42473577202e1ebdb5971a44669225a24260c390480bc850"
Jul 17 13:14:33.305459 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:14:33.305396    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc_odh-ai-gateway-infra(282b8ba5-c7e4-4e83-a683-4e96c1dec6fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc" podUID="282b8ba5-c7e4-4e83-a683-4e96c1dec6fd"
Jul 17 13:14:33.306862 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:33.306839    2569 generic.go:356] "Generic (PLEG): container finished" podID="97b459ab-0b13-4d1d-aabd-03c031865ea5" containerID="71c7523e7c8c78f3e116ea687efd4927c7ad0f08b9d09ec9191469bca7417ea5" exitCode=6
Jul 17 13:14:33.306982 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:33.306917    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j" event={"ID":"97b459ab-0b13-4d1d-aabd-03c031865ea5","Type":"ContainerDied","Data":"71c7523e7c8c78f3e116ea687efd4927c7ad0f08b9d09ec9191469bca7417ea5"}
Jul 17 13:14:33.306982 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:33.306970    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j" event={"ID":"97b459ab-0b13-4d1d-aabd-03c031865ea5","Type":"ContainerStarted","Data":"406eadec55529ccdbb39aef3704b12193f32b3d206f94cf9079fce2b73ac740e"}
Jul 17 13:14:33.307060 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:33.307043    2569 scope.go:117] "RemoveContainer" containerID="71c7523e7c8c78f3e116ea687efd4927c7ad0f08b9d09ec9191469bca7417ea5"
Jul 17 13:14:34.313088 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:34.313030    2569 generic.go:356] "Generic (PLEG): container finished" podID="97b459ab-0b13-4d1d-aabd-03c031865ea5" containerID="9c9af826913beb9b3633d718a4def1386a37f9341b1982b20689db6c426cb058" exitCode=6
Jul 17 13:14:34.313429 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:34.313112    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j" event={"ID":"97b459ab-0b13-4d1d-aabd-03c031865ea5","Type":"ContainerDied","Data":"9c9af826913beb9b3633d718a4def1386a37f9341b1982b20689db6c426cb058"}
Jul 17 13:14:34.313429 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:34.313172    2569 scope.go:117] "RemoveContainer" containerID="71c7523e7c8c78f3e116ea687efd4927c7ad0f08b9d09ec9191469bca7417ea5"
Jul 17 13:14:34.313429 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:34.313283    2569 scope.go:117] "RemoveContainer" containerID="9c9af826913beb9b3633d718a4def1386a37f9341b1982b20689db6c426cb058"
Jul 17 13:14:34.313562 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:14:34.313476    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j_odh-ai-gateway-infra(97b459ab-0b13-4d1d-aabd-03c031865ea5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j" podUID="97b459ab-0b13-4d1d-aabd-03c031865ea5"
Jul 17 13:14:34.314986 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:34.314967    2569 scope.go:117] "RemoveContainer" containerID="57f1c799303cdb1e42473577202e1ebdb5971a44669225a24260c390480bc850"
Jul 17 13:14:34.315116 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:14:34.315100    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc_odh-ai-gateway-infra(282b8ba5-c7e4-4e83-a683-4e96c1dec6fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc" podUID="282b8ba5-c7e4-4e83-a683-4e96c1dec6fd"
Jul 17 13:14:35.321826 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:35.321801    2569 scope.go:117] "RemoveContainer" containerID="9c9af826913beb9b3633d718a4def1386a37f9341b1982b20689db6c426cb058"
Jul 17 13:14:35.322219 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:14:35.321993    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j_odh-ai-gateway-infra(97b459ab-0b13-4d1d-aabd-03c031865ea5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j" podUID="97b459ab-0b13-4d1d-aabd-03c031865ea5"
Jul 17 13:14:36.628704 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:36.628674    2569 scope.go:117] "RemoveContainer" containerID="7df4453a14088c4e66a4ac43e075b38ae22ef62802442a5957693e34e346982c"
Jul 17 13:14:36.629142 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:14:36.628882    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:14:47.629032 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:47.629001    2569 scope.go:117] "RemoveContainer" containerID="57f1c799303cdb1e42473577202e1ebdb5971a44669225a24260c390480bc850"
Jul 17 13:14:47.629516 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:47.629156    2569 scope.go:117] "RemoveContainer" containerID="7df4453a14088c4e66a4ac43e075b38ae22ef62802442a5957693e34e346982c"
Jul 17 13:14:47.629516 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:14:47.629304    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:14:48.384122 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:48.384085    2569 generic.go:356] "Generic (PLEG): container finished" podID="282b8ba5-c7e4-4e83-a683-4e96c1dec6fd" containerID="74bfbd188961fd0fef4dd3f215a3b4e85f400c69d876805dbec923785b7fd460" exitCode=6
Jul 17 13:14:48.384250 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:48.384160    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc" event={"ID":"282b8ba5-c7e4-4e83-a683-4e96c1dec6fd","Type":"ContainerDied","Data":"74bfbd188961fd0fef4dd3f215a3b4e85f400c69d876805dbec923785b7fd460"}
Jul 17 13:14:48.384250 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:48.384227    2569 scope.go:117] "RemoveContainer" containerID="57f1c799303cdb1e42473577202e1ebdb5971a44669225a24260c390480bc850"
Jul 17 13:14:48.384511 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:48.384489    2569 scope.go:117] "RemoveContainer" containerID="74bfbd188961fd0fef4dd3f215a3b4e85f400c69d876805dbec923785b7fd460"
Jul 17 13:14:48.384707 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:14:48.384684    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc_odh-ai-gateway-infra(282b8ba5-c7e4-4e83-a683-4e96c1dec6fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc" podUID="282b8ba5-c7e4-4e83-a683-4e96c1dec6fd"
Jul 17 13:14:48.628567 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:48.628514    2569 scope.go:117] "RemoveContainer" containerID="9c9af826913beb9b3633d718a4def1386a37f9341b1982b20689db6c426cb058"
Jul 17 13:14:49.390657 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:49.390628    2569 generic.go:356] "Generic (PLEG): container finished" podID="97b459ab-0b13-4d1d-aabd-03c031865ea5" containerID="8aced7b45941158d8f21241bb3831d487d98457641bfe5eba9b8d5b85df0589d" exitCode=6
Jul 17 13:14:49.391022 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:49.390701    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j" event={"ID":"97b459ab-0b13-4d1d-aabd-03c031865ea5","Type":"ContainerDied","Data":"8aced7b45941158d8f21241bb3831d487d98457641bfe5eba9b8d5b85df0589d"}
Jul 17 13:14:49.391022 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:49.390768    2569 scope.go:117] "RemoveContainer" containerID="9c9af826913beb9b3633d718a4def1386a37f9341b1982b20689db6c426cb058"
Jul 17 13:14:49.391135 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:49.391056    2569 scope.go:117] "RemoveContainer" containerID="8aced7b45941158d8f21241bb3831d487d98457641bfe5eba9b8d5b85df0589d"
Jul 17 13:14:49.391252 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:14:49.391230    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j_odh-ai-gateway-infra(97b459ab-0b13-4d1d-aabd-03c031865ea5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j" podUID="97b459ab-0b13-4d1d-aabd-03c031865ea5"
Jul 17 13:14:49.414225 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:49.414202    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc"]
Jul 17 13:14:49.545487 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:49.545464    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc"
Jul 17 13:14:49.562715 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:49.562690    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/282b8ba5-c7e4-4e83-a683-4e96c1dec6fd-maas-api-service-ca\") pod \"282b8ba5-c7e4-4e83-a683-4e96c1dec6fd\" (UID: \"282b8ba5-c7e4-4e83-a683-4e96c1dec6fd\") "
Jul 17 13:14:49.563017 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:49.562996    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/282b8ba5-c7e4-4e83-a683-4e96c1dec6fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "282b8ba5-c7e4-4e83-a683-4e96c1dec6fd" (UID: "282b8ba5-c7e4-4e83-a683-4e96c1dec6fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:14:49.663360 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:49.663337    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/282b8ba5-c7e4-4e83-a683-4e96c1dec6fd-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:14:50.399424 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:50.399391    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc"
Jul 17 13:14:50.399897 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:50.399436    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc" event={"ID":"282b8ba5-c7e4-4e83-a683-4e96c1dec6fd","Type":"ContainerDied","Data":"4cedfa7504abb459a65bc1cef8467d0ac8a4f3cf8f7cfde193ab8daf043415d3"}
Jul 17 13:14:50.399897 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:50.399524    2569 scope.go:117] "RemoveContainer" containerID="74bfbd188961fd0fef4dd3f215a3b4e85f400c69d876805dbec923785b7fd460"
Jul 17 13:14:50.419191 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:50.419158    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc"]
Jul 17 13:14:50.425141 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:50.425118    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j"]
Jul 17 13:14:50.428234 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:50.428209    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tvcjc"]
Jul 17 13:14:50.558001 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:50.557976    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j"
Jul 17 13:14:50.571723 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:50.571704    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b459ab-0b13-4d1d-aabd-03c031865ea5-maas-api-service-ca\") pod \"97b459ab-0b13-4d1d-aabd-03c031865ea5\" (UID: \"97b459ab-0b13-4d1d-aabd-03c031865ea5\") "
Jul 17 13:14:50.572026 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:50.572006    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b459ab-0b13-4d1d-aabd-03c031865ea5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "97b459ab-0b13-4d1d-aabd-03c031865ea5" (UID: "97b459ab-0b13-4d1d-aabd-03c031865ea5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:14:50.672881 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:50.672818    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97b459ab-0b13-4d1d-aabd-03c031865ea5-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:14:51.409060 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:51.409037    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j"
Jul 17 13:14:51.409471 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:51.409036    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j" event={"ID":"97b459ab-0b13-4d1d-aabd-03c031865ea5","Type":"ContainerDied","Data":"406eadec55529ccdbb39aef3704b12193f32b3d206f94cf9079fce2b73ac740e"}
Jul 17 13:14:51.409471 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:51.409180    2569 scope.go:117] "RemoveContainer" containerID="8aced7b45941158d8f21241bb3831d487d98457641bfe5eba9b8d5b85df0589d"
Jul 17 13:14:51.432679 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:51.432652    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j"]
Jul 17 13:14:51.440766 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:51.440730    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-v4r4j"]
Jul 17 13:14:51.634165 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:51.634137    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="282b8ba5-c7e4-4e83-a683-4e96c1dec6fd" path="/var/lib/kubelet/pods/282b8ba5-c7e4-4e83-a683-4e96c1dec6fd/volumes"
Jul 17 13:14:51.634479 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:14:51.634464    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97b459ab-0b13-4d1d-aabd-03c031865ea5" path="/var/lib/kubelet/pods/97b459ab-0b13-4d1d-aabd-03c031865ea5/volumes"
Jul 17 13:15:00.144107 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.144069    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt"]
Jul 17 13:15:00.144572 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.144542    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="282b8ba5-c7e4-4e83-a683-4e96c1dec6fd" containerName="revoke-keys"
Jul 17 13:15:00.144572 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.144561    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="282b8ba5-c7e4-4e83-a683-4e96c1dec6fd" containerName="revoke-keys"
Jul 17 13:15:00.144689 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.144590    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="282b8ba5-c7e4-4e83-a683-4e96c1dec6fd" containerName="revoke-keys"
Jul 17 13:15:00.144689 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.144599    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="282b8ba5-c7e4-4e83-a683-4e96c1dec6fd" containerName="revoke-keys"
Jul 17 13:15:00.144689 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.144609    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b459ab-0b13-4d1d-aabd-03c031865ea5" containerName="revoke-keys"
Jul 17 13:15:00.144689 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.144618    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b459ab-0b13-4d1d-aabd-03c031865ea5" containerName="revoke-keys"
Jul 17 13:15:00.144689 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.144630    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b459ab-0b13-4d1d-aabd-03c031865ea5" containerName="revoke-keys"
Jul 17 13:15:00.144689 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.144639    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b459ab-0b13-4d1d-aabd-03c031865ea5" containerName="revoke-keys"
Jul 17 13:15:00.144689 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.144663    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b459ab-0b13-4d1d-aabd-03c031865ea5" containerName="revoke-keys"
Jul 17 13:15:00.144689 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.144671    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b459ab-0b13-4d1d-aabd-03c031865ea5" containerName="revoke-keys"
Jul 17 13:15:00.145138 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.144790    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="282b8ba5-c7e4-4e83-a683-4e96c1dec6fd" containerName="revoke-keys"
Jul 17 13:15:00.145138 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.144804    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="97b459ab-0b13-4d1d-aabd-03c031865ea5" containerName="revoke-keys"
Jul 17 13:15:00.145138 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.144812    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="282b8ba5-c7e4-4e83-a683-4e96c1dec6fd" containerName="revoke-keys"
Jul 17 13:15:00.145138 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.144827    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="97b459ab-0b13-4d1d-aabd-03c031865ea5" containerName="revoke-keys"
Jul 17 13:15:00.149285 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.149262    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738235-fk7wd"]
Jul 17 13:15:00.149432 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.149381    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt"
Jul 17 13:15:00.149639 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.149622    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="282b8ba5-c7e4-4e83-a683-4e96c1dec6fd" containerName="revoke-keys"
Jul 17 13:15:00.149708 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.149641    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="282b8ba5-c7e4-4e83-a683-4e96c1dec6fd" containerName="revoke-keys"
Jul 17 13:15:00.149813 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.149799    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="97b459ab-0b13-4d1d-aabd-03c031865ea5" containerName="revoke-keys"
Jul 17 13:15:00.149885 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.149814    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="282b8ba5-c7e4-4e83-a683-4e96c1dec6fd" containerName="revoke-keys"
Jul 17 13:15:00.152510 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.152488    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738235-fk7wd"
Jul 17 13:15:00.152795 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.152781    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:15:00.155236 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.155158    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt"]
Jul 17 13:15:00.158563 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.157936    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738235-fk7wd"]
Jul 17 13:15:00.162670 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.162650    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt"
Jul 17 13:15:00.170351 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.170334    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738235-fk7wd"
Jul 17 13:15:00.326836 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.326773    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738235-fk7wd"]
Jul 17 13:15:00.329052 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:15:00.329025    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c391ea4_c0bd_4ab0_bc86_2fa99f79ce50.slice/crio-e5243d65715e7e5140da0616a7a66957abf42ad0ee9cb3edb4c474895879923d WatchSource:0}: Error finding container e5243d65715e7e5140da0616a7a66957abf42ad0ee9cb3edb4c474895879923d: Status 404 returned error can't find the container with id e5243d65715e7e5140da0616a7a66957abf42ad0ee9cb3edb4c474895879923d
Jul 17 13:15:00.454801 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.454735    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738235-fk7wd" event={"ID":"8c391ea4-c0bd-4ab0-bc86-2fa99f79ce50","Type":"ContainerStarted","Data":"e5243d65715e7e5140da0616a7a66957abf42ad0ee9cb3edb4c474895879923d"}
Jul 17 13:15:00.517489 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.517466    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt"]
Jul 17 13:15:00.520022 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:15:00.519985    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a455d37_5007_4c62_a0de_e70713db4dfa.slice/crio-95b5a5e3332bfd1d34e09b2dbf7a1b1d2ace7003716705824b18272092b2c580 WatchSource:0}: Error finding container 95b5a5e3332bfd1d34e09b2dbf7a1b1d2ace7003716705824b18272092b2c580: Status 404 returned error can't find the container with id 95b5a5e3332bfd1d34e09b2dbf7a1b1d2ace7003716705824b18272092b2c580
Jul 17 13:15:00.628597 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:00.628576    2569 scope.go:117] "RemoveContainer" containerID="7df4453a14088c4e66a4ac43e075b38ae22ef62802442a5957693e34e346982c"
Jul 17 13:15:01.461506 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:01.461481    2569 generic.go:356] "Generic (PLEG): container finished" podID="4a455d37-5007-4c62-a0de-e70713db4dfa" containerID="113b6da9b3ed366bfd3e3e79bbe66d6837ea3caa3e5fa475e75a94b90d5ae6ae" exitCode=7
Jul 17 13:15:01.461904 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:01.461559    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt" event={"ID":"4a455d37-5007-4c62-a0de-e70713db4dfa","Type":"ContainerDied","Data":"113b6da9b3ed366bfd3e3e79bbe66d6837ea3caa3e5fa475e75a94b90d5ae6ae"}
Jul 17 13:15:01.461904 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:01.461612    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt" event={"ID":"4a455d37-5007-4c62-a0de-e70713db4dfa","Type":"ContainerStarted","Data":"95b5a5e3332bfd1d34e09b2dbf7a1b1d2ace7003716705824b18272092b2c580"}
Jul 17 13:15:01.461904 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:01.461724    2569 scope.go:117] "RemoveContainer" containerID="113b6da9b3ed366bfd3e3e79bbe66d6837ea3caa3e5fa475e75a94b90d5ae6ae"
Jul 17 13:15:01.463174 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:01.463155    2569 generic.go:356] "Generic (PLEG): container finished" podID="8c391ea4-c0bd-4ab0-bc86-2fa99f79ce50" containerID="be376345e5d317e784c5ae444a501d149e21ad500682f48eb4cd4f250605e335" exitCode=0
Jul 17 13:15:01.463279 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:01.463235    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738235-fk7wd" event={"ID":"8c391ea4-c0bd-4ab0-bc86-2fa99f79ce50","Type":"ContainerDied","Data":"be376345e5d317e784c5ae444a501d149e21ad500682f48eb4cd4f250605e335"}
Jul 17 13:15:02.469907 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:02.469841    2569 generic.go:356] "Generic (PLEG): container finished" podID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerID="8bbce74974119b0104ff37782f24cebc82f73263db21b95299f5ee3a04cca985" exitCode=1
Jul 17 13:15:02.470281 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:02.469919    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" event={"ID":"2c1b598d-1c76-4afd-8c32-c50b57a701c5","Type":"ContainerDied","Data":"8bbce74974119b0104ff37782f24cebc82f73263db21b95299f5ee3a04cca985"}
Jul 17 13:15:02.470281 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:02.469986    2569 scope.go:117] "RemoveContainer" containerID="7df4453a14088c4e66a4ac43e075b38ae22ef62802442a5957693e34e346982c"
Jul 17 13:15:02.470398 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:02.470348    2569 scope.go:117] "RemoveContainer" containerID="8bbce74974119b0104ff37782f24cebc82f73263db21b95299f5ee3a04cca985"
Jul 17 13:15:02.470614 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:02.470581    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:15:02.472100 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:02.472076    2569 generic.go:356] "Generic (PLEG): container finished" podID="4a455d37-5007-4c62-a0de-e70713db4dfa" containerID="30fa20515d2f667b97dca995ff3768c79a7336db5016c3e3fd3bb9d3a741c1c4" exitCode=7
Jul 17 13:15:02.472220 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:02.472160    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt" event={"ID":"4a455d37-5007-4c62-a0de-e70713db4dfa","Type":"ContainerDied","Data":"30fa20515d2f667b97dca995ff3768c79a7336db5016c3e3fd3bb9d3a741c1c4"}
Jul 17 13:15:02.472362 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:02.472346    2569 scope.go:117] "RemoveContainer" containerID="30fa20515d2f667b97dca995ff3768c79a7336db5016c3e3fd3bb9d3a741c1c4"
Jul 17 13:15:02.472637 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:02.472534    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt_odh-ai-gateway-infra(4a455d37-5007-4c62-a0de-e70713db4dfa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt" podUID="4a455d37-5007-4c62-a0de-e70713db4dfa"
Jul 17 13:15:02.481973 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:02.481952    2569 scope.go:117] "RemoveContainer" containerID="113b6da9b3ed366bfd3e3e79bbe66d6837ea3caa3e5fa475e75a94b90d5ae6ae"
Jul 17 13:15:02.607274 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:02.607245    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738235-fk7wd"
Jul 17 13:15:03.479275 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:03.479246    2569 scope.go:117] "RemoveContainer" containerID="30fa20515d2f667b97dca995ff3768c79a7336db5016c3e3fd3bb9d3a741c1c4"
Jul 17 13:15:03.479782 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:03.479472    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt_odh-ai-gateway-infra(4a455d37-5007-4c62-a0de-e70713db4dfa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt" podUID="4a455d37-5007-4c62-a0de-e70713db4dfa"
Jul 17 13:15:03.480619 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:03.480595    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738235-fk7wd"
Jul 17 13:15:03.480725 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:03.480620    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738235-fk7wd" event={"ID":"8c391ea4-c0bd-4ab0-bc86-2fa99f79ce50","Type":"ContainerDied","Data":"e5243d65715e7e5140da0616a7a66957abf42ad0ee9cb3edb4c474895879923d"}
Jul 17 13:15:03.480725 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:03.480678    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5243d65715e7e5140da0616a7a66957abf42ad0ee9cb3edb4c474895879923d"
Jul 17 13:15:04.172358 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:04.172325    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:15:04.172716 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:04.172697    2569 scope.go:117] "RemoveContainer" containerID="8bbce74974119b0104ff37782f24cebc82f73263db21b95299f5ee3a04cca985"
Jul 17 13:15:04.172948 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:04.172924    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:15:04.232726 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:04.232693    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:15:04.487134 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:04.487096    2569 scope.go:117] "RemoveContainer" containerID="8bbce74974119b0104ff37782f24cebc82f73263db21b95299f5ee3a04cca985"
Jul 17 13:15:04.487529 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:04.487312    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:15:15.630671 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:15.630636    2569 scope.go:117] "RemoveContainer" containerID="30fa20515d2f667b97dca995ff3768c79a7336db5016c3e3fd3bb9d3a741c1c4"
Jul 17 13:15:16.544286 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:16.544256    2569 generic.go:356] "Generic (PLEG): container finished" podID="4a455d37-5007-4c62-a0de-e70713db4dfa" containerID="7ed6aa303506d4757b600173040033cd8c774ee6a44656ae7a3fea97d4ff62b2" exitCode=7
Jul 17 13:15:16.544454 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:16.544324    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt" event={"ID":"4a455d37-5007-4c62-a0de-e70713db4dfa","Type":"ContainerDied","Data":"7ed6aa303506d4757b600173040033cd8c774ee6a44656ae7a3fea97d4ff62b2"}
Jul 17 13:15:16.544454 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:16.544386    2569 scope.go:117] "RemoveContainer" containerID="30fa20515d2f667b97dca995ff3768c79a7336db5016c3e3fd3bb9d3a741c1c4"
Jul 17 13:15:16.544664 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:16.544637    2569 scope.go:117] "RemoveContainer" containerID="7ed6aa303506d4757b600173040033cd8c774ee6a44656ae7a3fea97d4ff62b2"
Jul 17 13:15:16.544879 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:16.544851    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt_odh-ai-gateway-infra(4a455d37-5007-4c62-a0de-e70713db4dfa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt" podUID="4a455d37-5007-4c62-a0de-e70713db4dfa"
Jul 17 13:15:17.603800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:17.603746    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt"]
Jul 17 13:15:17.628294 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:17.628272    2569 scope.go:117] "RemoveContainer" containerID="8bbce74974119b0104ff37782f24cebc82f73263db21b95299f5ee3a04cca985"
Jul 17 13:15:17.628457 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:17.628438    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:15:17.736959 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:17.736937    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt"
Jul 17 13:15:18.556916 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:18.556888    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt"
Jul 17 13:15:18.557090 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:18.556879    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt" event={"ID":"4a455d37-5007-4c62-a0de-e70713db4dfa","Type":"ContainerDied","Data":"95b5a5e3332bfd1d34e09b2dbf7a1b1d2ace7003716705824b18272092b2c580"}
Jul 17 13:15:18.557090 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:18.557069    2569 scope.go:117] "RemoveContainer" containerID="7ed6aa303506d4757b600173040033cd8c774ee6a44656ae7a3fea97d4ff62b2"
Jul 17 13:15:18.585648 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:18.585623    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt"]
Jul 17 13:15:18.589455 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:18.589433    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-661ee8a2-29738235-dslvt"]
Jul 17 13:15:19.633388 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:19.633355    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a455d37-5007-4c62-a0de-e70713db4dfa" path="/var/lib/kubelet/pods/4a455d37-5007-4c62-a0de-e70713db4dfa/volumes"
Jul 17 13:15:20.813891 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:20.813858    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2"]
Jul 17 13:15:20.814285 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:20.814185    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a455d37-5007-4c62-a0de-e70713db4dfa" containerName="cleanup"
Jul 17 13:15:20.814285 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:20.814195    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a455d37-5007-4c62-a0de-e70713db4dfa" containerName="cleanup"
Jul 17 13:15:20.814285 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:20.814203    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c391ea4-c0bd-4ab0-bc86-2fa99f79ce50" containerName="cleanup"
Jul 17 13:15:20.814285 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:20.814208    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c391ea4-c0bd-4ab0-bc86-2fa99f79ce50" containerName="cleanup"
Jul 17 13:15:20.814285 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:20.814229    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a455d37-5007-4c62-a0de-e70713db4dfa" containerName="cleanup"
Jul 17 13:15:20.814285 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:20.814234    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a455d37-5007-4c62-a0de-e70713db4dfa" containerName="cleanup"
Jul 17 13:15:20.814285 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:20.814285    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c391ea4-c0bd-4ab0-bc86-2fa99f79ce50" containerName="cleanup"
Jul 17 13:15:20.814550 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:20.814292    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a455d37-5007-4c62-a0de-e70713db4dfa" containerName="cleanup"
Jul 17 13:15:20.814550 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:20.814302    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a455d37-5007-4c62-a0de-e70713db4dfa" containerName="cleanup"
Jul 17 13:15:20.818271 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:20.818254    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2"
Jul 17 13:15:20.821400 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:20.821381    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:15:20.831355 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:20.831329    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2"]
Jul 17 13:15:20.901263 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:20.901238    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802eba4e-9187-425a-a189-3d5403ad7d6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2\" (UID: \"802eba4e-9187-425a-a189-3d5403ad7d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2"
Jul 17 13:15:21.001915 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:21.001891    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802eba4e-9187-425a-a189-3d5403ad7d6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2\" (UID: \"802eba4e-9187-425a-a189-3d5403ad7d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2"
Jul 17 13:15:21.002454 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:21.002435    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802eba4e-9187-425a-a189-3d5403ad7d6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2\" (UID: \"802eba4e-9187-425a-a189-3d5403ad7d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2"
Jul 17 13:15:21.127533 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:21.127479    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2"
Jul 17 13:15:21.268990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:21.268961    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2"]
Jul 17 13:15:21.272441 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:15:21.272413    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802eba4e_9187_425a_a189_3d5403ad7d6d.slice/crio-83ad52590a10fed0347a54dfbef52ff1dcd8bfc71ce26c063160c8961a624cf9 WatchSource:0}: Error finding container 83ad52590a10fed0347a54dfbef52ff1dcd8bfc71ce26c063160c8961a624cf9: Status 404 returned error can't find the container with id 83ad52590a10fed0347a54dfbef52ff1dcd8bfc71ce26c063160c8961a624cf9
Jul 17 13:15:21.573550 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:21.573523    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2" event={"ID":"802eba4e-9187-425a-a189-3d5403ad7d6d","Type":"ContainerStarted","Data":"4b6d833d5b65522a8bf1eebb3cbf5ae58d99bf12cb8fdcd3065b93b0c92d0ba0"}
Jul 17 13:15:21.573645 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:21.573570    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2" event={"ID":"802eba4e-9187-425a-a189-3d5403ad7d6d","Type":"ContainerStarted","Data":"83ad52590a10fed0347a54dfbef52ff1dcd8bfc71ce26c063160c8961a624cf9"}
Jul 17 13:15:21.595714 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:21.595673    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2" podStartSLOduration=1.595656897 podStartE2EDuration="1.595656897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:15:21.593827377 +0000 UTC m=+3196.597874259" watchObservedRunningTime="2026-07-17 13:15:21.595656897 +0000 UTC m=+3196.599703779"
Jul 17 13:15:21.989625 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:21.989601    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f"]
Jul 17 13:15:21.989965 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:21.989950    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a455d37-5007-4c62-a0de-e70713db4dfa" containerName="cleanup"
Jul 17 13:15:21.990005 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:21.989967    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a455d37-5007-4c62-a0de-e70713db4dfa" containerName="cleanup"
Jul 17 13:15:21.990041 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:21.990034    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a455d37-5007-4c62-a0de-e70713db4dfa" containerName="cleanup"
Jul 17 13:15:21.992686 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:21.992670    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f"
Jul 17 13:15:22.010215 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:22.010194    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f"]
Jul 17 13:15:22.111438 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:22.111380    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd2595b-e2a9-4277-91f2-e99519bca86e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f\" (UID: \"0cd2595b-e2a9-4277-91f2-e99519bca86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f"
Jul 17 13:15:22.212705 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:22.212682    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd2595b-e2a9-4277-91f2-e99519bca86e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f\" (UID: \"0cd2595b-e2a9-4277-91f2-e99519bca86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f"
Jul 17 13:15:22.213277 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:22.213258    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd2595b-e2a9-4277-91f2-e99519bca86e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f\" (UID: \"0cd2595b-e2a9-4277-91f2-e99519bca86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f"
Jul 17 13:15:22.301810 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:22.301789    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f"
Jul 17 13:15:22.467776 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:22.467736    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f"]
Jul 17 13:15:22.470339 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:15:22.470310    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd2595b_e2a9_4277_91f2_e99519bca86e.slice/crio-a24a660028abfd1cf0d3faa2d0d84069638ae62e07429c8ef14b6226405eeb4c WatchSource:0}: Error finding container a24a660028abfd1cf0d3faa2d0d84069638ae62e07429c8ef14b6226405eeb4c: Status 404 returned error can't find the container with id a24a660028abfd1cf0d3faa2d0d84069638ae62e07429c8ef14b6226405eeb4c
Jul 17 13:15:22.578681 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:22.578656    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f" event={"ID":"0cd2595b-e2a9-4277-91f2-e99519bca86e","Type":"ContainerStarted","Data":"a24a660028abfd1cf0d3faa2d0d84069638ae62e07429c8ef14b6226405eeb4c"}
Jul 17 13:15:22.580095 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:22.580071    2569 generic.go:356] "Generic (PLEG): container finished" podID="802eba4e-9187-425a-a189-3d5403ad7d6d" containerID="4b6d833d5b65522a8bf1eebb3cbf5ae58d99bf12cb8fdcd3065b93b0c92d0ba0" exitCode=6
Jul 17 13:15:22.580170 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:22.580144    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2" event={"ID":"802eba4e-9187-425a-a189-3d5403ad7d6d","Type":"ContainerDied","Data":"4b6d833d5b65522a8bf1eebb3cbf5ae58d99bf12cb8fdcd3065b93b0c92d0ba0"}
Jul 17 13:15:22.580287 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:22.580274    2569 scope.go:117] "RemoveContainer" containerID="4b6d833d5b65522a8bf1eebb3cbf5ae58d99bf12cb8fdcd3065b93b0c92d0ba0"
Jul 17 13:15:23.148480 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:23.148454    2569 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802eba4e_9187_425a_a189_3d5403ad7d6d.slice/crio-8cbc7310731fa8d3ebd62e3a22839d02921398cbdb46b313751d9649ec3ed578.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 13:15:23.592608 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:23.592542    2569 generic.go:356] "Generic (PLEG): container finished" podID="0cd2595b-e2a9-4277-91f2-e99519bca86e" containerID="1bec92c295ac0bc5638f3bc5399cb18c5a32905c51d0d53c4671f33206779245" exitCode=6
Jul 17 13:15:23.592725 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:23.592624    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f" event={"ID":"0cd2595b-e2a9-4277-91f2-e99519bca86e","Type":"ContainerDied","Data":"1bec92c295ac0bc5638f3bc5399cb18c5a32905c51d0d53c4671f33206779245"}
Jul 17 13:15:23.592791 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:23.592781    2569 scope.go:117] "RemoveContainer" containerID="1bec92c295ac0bc5638f3bc5399cb18c5a32905c51d0d53c4671f33206779245"
Jul 17 13:15:23.594485 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:23.594464    2569 generic.go:356] "Generic (PLEG): container finished" podID="802eba4e-9187-425a-a189-3d5403ad7d6d" containerID="8cbc7310731fa8d3ebd62e3a22839d02921398cbdb46b313751d9649ec3ed578" exitCode=6
Jul 17 13:15:23.594574 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:23.594539    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2" event={"ID":"802eba4e-9187-425a-a189-3d5403ad7d6d","Type":"ContainerDied","Data":"8cbc7310731fa8d3ebd62e3a22839d02921398cbdb46b313751d9649ec3ed578"}
Jul 17 13:15:23.594613 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:23.594593    2569 scope.go:117] "RemoveContainer" containerID="4b6d833d5b65522a8bf1eebb3cbf5ae58d99bf12cb8fdcd3065b93b0c92d0ba0"
Jul 17 13:15:23.594689 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:23.594674    2569 scope.go:117] "RemoveContainer" containerID="8cbc7310731fa8d3ebd62e3a22839d02921398cbdb46b313751d9649ec3ed578"
Jul 17 13:15:23.595023 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:23.594871    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2_odh-ai-gateway-infra(802eba4e-9187-425a-a189-3d5403ad7d6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2" podUID="802eba4e-9187-425a-a189-3d5403ad7d6d"
Jul 17 13:15:24.601551 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:24.601519    2569 generic.go:356] "Generic (PLEG): container finished" podID="0cd2595b-e2a9-4277-91f2-e99519bca86e" containerID="e67fdc0ac7110ddb5296c03b57da0469c0e40ae0698fbe4ed8e60b6aa38c68b7" exitCode=6
Jul 17 13:15:24.601980 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:24.601606    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f" event={"ID":"0cd2595b-e2a9-4277-91f2-e99519bca86e","Type":"ContainerDied","Data":"e67fdc0ac7110ddb5296c03b57da0469c0e40ae0698fbe4ed8e60b6aa38c68b7"}
Jul 17 13:15:24.601980 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:24.601671    2569 scope.go:117] "RemoveContainer" containerID="1bec92c295ac0bc5638f3bc5399cb18c5a32905c51d0d53c4671f33206779245"
Jul 17 13:15:24.601980 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:24.601888    2569 scope.go:117] "RemoveContainer" containerID="e67fdc0ac7110ddb5296c03b57da0469c0e40ae0698fbe4ed8e60b6aa38c68b7"
Jul 17 13:15:24.602122 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:24.602076    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f_odh-ai-gateway-infra(0cd2595b-e2a9-4277-91f2-e99519bca86e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f" podUID="0cd2595b-e2a9-4277-91f2-e99519bca86e"
Jul 17 13:15:24.603799 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:24.603780    2569 scope.go:117] "RemoveContainer" containerID="8cbc7310731fa8d3ebd62e3a22839d02921398cbdb46b313751d9649ec3ed578"
Jul 17 13:15:24.603959 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:24.603940    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2_odh-ai-gateway-infra(802eba4e-9187-425a-a189-3d5403ad7d6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2" podUID="802eba4e-9187-425a-a189-3d5403ad7d6d"
Jul 17 13:15:25.609270 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:25.609246    2569 scope.go:117] "RemoveContainer" containerID="e67fdc0ac7110ddb5296c03b57da0469c0e40ae0698fbe4ed8e60b6aa38c68b7"
Jul 17 13:15:25.609630 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:25.609403    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f_odh-ai-gateway-infra(0cd2595b-e2a9-4277-91f2-e99519bca86e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f" podUID="0cd2595b-e2a9-4277-91f2-e99519bca86e"
Jul 17 13:15:28.308765 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:28.308725    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d"]
Jul 17 13:15:28.313453 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:28.313434    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d"
Jul 17 13:15:28.323516 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:28.323490    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d"]
Jul 17 13:15:28.462504 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:28.462472    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538b59c0-d6e0-4926-be5c-a831cfda6b74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d\" (UID: \"538b59c0-d6e0-4926-be5c-a831cfda6b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d"
Jul 17 13:15:28.563828 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:28.563742    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538b59c0-d6e0-4926-be5c-a831cfda6b74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d\" (UID: \"538b59c0-d6e0-4926-be5c-a831cfda6b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d"
Jul 17 13:15:28.564276 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:28.564259    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538b59c0-d6e0-4926-be5c-a831cfda6b74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d\" (UID: \"538b59c0-d6e0-4926-be5c-a831cfda6b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d"
Jul 17 13:15:28.624110 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:28.624086    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d"
Jul 17 13:15:28.977609 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:28.977583    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d"]
Jul 17 13:15:28.979145 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:15:28.979122    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538b59c0_d6e0_4926_be5c_a831cfda6b74.slice/crio-9636172b652368ade05a818c871a96f379a5811db5692fd669f71496276f7311 WatchSource:0}: Error finding container 9636172b652368ade05a818c871a96f379a5811db5692fd669f71496276f7311: Status 404 returned error can't find the container with id 9636172b652368ade05a818c871a96f379a5811db5692fd669f71496276f7311
Jul 17 13:15:29.628662 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:29.628632    2569 generic.go:356] "Generic (PLEG): container finished" podID="538b59c0-d6e0-4926-be5c-a831cfda6b74" containerID="9708543411c6d18574eaee969c457e5cb4a41ae23af8fc5331904a93cdc970c9" exitCode=7
Jul 17 13:15:29.629043 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:29.628917    2569 scope.go:117] "RemoveContainer" containerID="9708543411c6d18574eaee969c457e5cb4a41ae23af8fc5331904a93cdc970c9"
Jul 17 13:15:29.633321 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:29.633294    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d" event={"ID":"538b59c0-d6e0-4926-be5c-a831cfda6b74","Type":"ContainerDied","Data":"9708543411c6d18574eaee969c457e5cb4a41ae23af8fc5331904a93cdc970c9"}
Jul 17 13:15:29.633420 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:29.633348    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d" event={"ID":"538b59c0-d6e0-4926-be5c-a831cfda6b74","Type":"ContainerStarted","Data":"9636172b652368ade05a818c871a96f379a5811db5692fd669f71496276f7311"}
Jul 17 13:15:30.628122 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:30.628091    2569 scope.go:117] "RemoveContainer" containerID="8bbce74974119b0104ff37782f24cebc82f73263db21b95299f5ee3a04cca985"
Jul 17 13:15:30.628266 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:30.628245    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:15:30.635450 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:30.635428    2569 generic.go:356] "Generic (PLEG): container finished" podID="538b59c0-d6e0-4926-be5c-a831cfda6b74" containerID="cd7044e0b15566ca75b981fcfc7e680c1d8949bb498cfd182c439cec8c5dc355" exitCode=7
Jul 17 13:15:30.635837 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:30.635510    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d" event={"ID":"538b59c0-d6e0-4926-be5c-a831cfda6b74","Type":"ContainerDied","Data":"cd7044e0b15566ca75b981fcfc7e680c1d8949bb498cfd182c439cec8c5dc355"}
Jul 17 13:15:30.635837 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:30.635569    2569 scope.go:117] "RemoveContainer" containerID="9708543411c6d18574eaee969c457e5cb4a41ae23af8fc5331904a93cdc970c9"
Jul 17 13:15:30.635837 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:30.635744    2569 scope.go:117] "RemoveContainer" containerID="cd7044e0b15566ca75b981fcfc7e680c1d8949bb498cfd182c439cec8c5dc355"
Jul 17 13:15:30.635985 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:30.635963    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d_odh-ai-gateway-infra(538b59c0-d6e0-4926-be5c-a831cfda6b74)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d" podUID="538b59c0-d6e0-4926-be5c-a831cfda6b74"
Jul 17 13:15:31.642159 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:31.642129    2569 scope.go:117] "RemoveContainer" containerID="cd7044e0b15566ca75b981fcfc7e680c1d8949bb498cfd182c439cec8c5dc355"
Jul 17 13:15:31.642572 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:31.642320    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d_odh-ai-gateway-infra(538b59c0-d6e0-4926-be5c-a831cfda6b74)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d" podUID="538b59c0-d6e0-4926-be5c-a831cfda6b74"
Jul 17 13:15:37.629225 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:37.629167    2569 scope.go:117] "RemoveContainer" containerID="8cbc7310731fa8d3ebd62e3a22839d02921398cbdb46b313751d9649ec3ed578"
Jul 17 13:15:37.629225 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:37.629224    2569 scope.go:117] "RemoveContainer" containerID="e67fdc0ac7110ddb5296c03b57da0469c0e40ae0698fbe4ed8e60b6aa38c68b7"
Jul 17 13:15:38.678964 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:38.678895    2569 generic.go:356] "Generic (PLEG): container finished" podID="0cd2595b-e2a9-4277-91f2-e99519bca86e" containerID="454c6b84895fbfac9f533772ab5d36f5854e4b08374b8b50e1d8d45d5b07ba55" exitCode=6
Jul 17 13:15:38.679350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:38.678970    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f" event={"ID":"0cd2595b-e2a9-4277-91f2-e99519bca86e","Type":"ContainerDied","Data":"454c6b84895fbfac9f533772ab5d36f5854e4b08374b8b50e1d8d45d5b07ba55"}
Jul 17 13:15:38.679350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:38.679025    2569 scope.go:117] "RemoveContainer" containerID="e67fdc0ac7110ddb5296c03b57da0469c0e40ae0698fbe4ed8e60b6aa38c68b7"
Jul 17 13:15:38.679350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:38.679284    2569 scope.go:117] "RemoveContainer" containerID="454c6b84895fbfac9f533772ab5d36f5854e4b08374b8b50e1d8d45d5b07ba55"
Jul 17 13:15:38.679491 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:38.679472    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f_odh-ai-gateway-infra(0cd2595b-e2a9-4277-91f2-e99519bca86e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f" podUID="0cd2595b-e2a9-4277-91f2-e99519bca86e"
Jul 17 13:15:38.681173 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:38.681152    2569 generic.go:356] "Generic (PLEG): container finished" podID="802eba4e-9187-425a-a189-3d5403ad7d6d" containerID="a79f6d10f0db1b3b2f1c9a1eb7596fa3af4522a206598bd3bbd50bed0d5f184c" exitCode=6
Jul 17 13:15:38.681275 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:38.681220    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2" event={"ID":"802eba4e-9187-425a-a189-3d5403ad7d6d","Type":"ContainerDied","Data":"a79f6d10f0db1b3b2f1c9a1eb7596fa3af4522a206598bd3bbd50bed0d5f184c"}
Jul 17 13:15:38.681488 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:38.681472    2569 scope.go:117] "RemoveContainer" containerID="a79f6d10f0db1b3b2f1c9a1eb7596fa3af4522a206598bd3bbd50bed0d5f184c"
Jul 17 13:15:38.681652 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:38.681635    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2_odh-ai-gateway-infra(802eba4e-9187-425a-a189-3d5403ad7d6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2" podUID="802eba4e-9187-425a-a189-3d5403ad7d6d"
Jul 17 13:15:38.689645 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:38.689625    2569 scope.go:117] "RemoveContainer" containerID="8cbc7310731fa8d3ebd62e3a22839d02921398cbdb46b313751d9649ec3ed578"
Jul 17 13:15:39.736573 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:39.736534    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f"]
Jul 17 13:15:39.768976 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:39.768944    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2"]
Jul 17 13:15:39.881878 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:39.881854    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f"
Jul 17 13:15:39.942117 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:39.942097    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2"
Jul 17 13:15:40.048524 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:40.048464    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd2595b-e2a9-4277-91f2-e99519bca86e-maas-api-service-ca\") pod \"0cd2595b-e2a9-4277-91f2-e99519bca86e\" (UID: \"0cd2595b-e2a9-4277-91f2-e99519bca86e\") "
Jul 17 13:15:40.048631 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:40.048565    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802eba4e-9187-425a-a189-3d5403ad7d6d-maas-api-service-ca\") pod \"802eba4e-9187-425a-a189-3d5403ad7d6d\" (UID: \"802eba4e-9187-425a-a189-3d5403ad7d6d\") "
Jul 17 13:15:40.048894 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:40.048870    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd2595b-e2a9-4277-91f2-e99519bca86e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0cd2595b-e2a9-4277-91f2-e99519bca86e" (UID: "0cd2595b-e2a9-4277-91f2-e99519bca86e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:15:40.048935 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:40.048911    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/802eba4e-9187-425a-a189-3d5403ad7d6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "802eba4e-9187-425a-a189-3d5403ad7d6d" (UID: "802eba4e-9187-425a-a189-3d5403ad7d6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:15:40.149975 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:40.149950    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802eba4e-9187-425a-a189-3d5403ad7d6d-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:15:40.149975 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:40.149973    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd2595b-e2a9-4277-91f2-e99519bca86e-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:15:40.695435 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:40.695406    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f"
Jul 17 13:15:40.695598 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:40.695456    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f" event={"ID":"0cd2595b-e2a9-4277-91f2-e99519bca86e","Type":"ContainerDied","Data":"a24a660028abfd1cf0d3faa2d0d84069638ae62e07429c8ef14b6226405eeb4c"}
Jul 17 13:15:40.695598 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:40.695523    2569 scope.go:117] "RemoveContainer" containerID="454c6b84895fbfac9f533772ab5d36f5854e4b08374b8b50e1d8d45d5b07ba55"
Jul 17 13:15:40.696948 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:40.696924    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2" event={"ID":"802eba4e-9187-425a-a189-3d5403ad7d6d","Type":"ContainerDied","Data":"83ad52590a10fed0347a54dfbef52ff1dcd8bfc71ce26c063160c8961a624cf9"}
Jul 17 13:15:40.697054 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:40.696960    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2"
Jul 17 13:15:40.704489 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:40.704468    2569 scope.go:117] "RemoveContainer" containerID="a79f6d10f0db1b3b2f1c9a1eb7596fa3af4522a206598bd3bbd50bed0d5f184c"
Jul 17 13:15:40.736453 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:40.736421    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f"]
Jul 17 13:15:40.748493 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:40.748465    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-vzv7f"]
Jul 17 13:15:40.789147 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:40.789126    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2"]
Jul 17 13:15:40.799417 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:40.799396    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q8tx2"]
Jul 17 13:15:41.634857 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:41.634829    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cd2595b-e2a9-4277-91f2-e99519bca86e" path="/var/lib/kubelet/pods/0cd2595b-e2a9-4277-91f2-e99519bca86e/volumes"
Jul 17 13:15:41.635187 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:41.635172    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="802eba4e-9187-425a-a189-3d5403ad7d6d" path="/var/lib/kubelet/pods/802eba4e-9187-425a-a189-3d5403ad7d6d/volumes"
Jul 17 13:15:42.628798 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:42.628770    2569 scope.go:117] "RemoveContainer" containerID="cd7044e0b15566ca75b981fcfc7e680c1d8949bb498cfd182c439cec8c5dc355"
Jul 17 13:15:43.628801 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:43.628778    2569 scope.go:117] "RemoveContainer" containerID="8bbce74974119b0104ff37782f24cebc82f73263db21b95299f5ee3a04cca985"
Jul 17 13:15:43.628974 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:43.628954    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:15:43.715014 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:43.714988    2569 generic.go:356] "Generic (PLEG): container finished" podID="538b59c0-d6e0-4926-be5c-a831cfda6b74" containerID="b976a3b11145b51fe40f71ca6e8c44a146a04e81b986bbd1f18f491b0d745f35" exitCode=7
Jul 17 13:15:43.715118 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:43.715040    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d" event={"ID":"538b59c0-d6e0-4926-be5c-a831cfda6b74","Type":"ContainerDied","Data":"b976a3b11145b51fe40f71ca6e8c44a146a04e81b986bbd1f18f491b0d745f35"}
Jul 17 13:15:43.715118 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:43.715095    2569 scope.go:117] "RemoveContainer" containerID="cd7044e0b15566ca75b981fcfc7e680c1d8949bb498cfd182c439cec8c5dc355"
Jul 17 13:15:43.715403 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:43.715375    2569 scope.go:117] "RemoveContainer" containerID="b976a3b11145b51fe40f71ca6e8c44a146a04e81b986bbd1f18f491b0d745f35"
Jul 17 13:15:43.715599 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:43.715572    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d_odh-ai-gateway-infra(538b59c0-d6e0-4926-be5c-a831cfda6b74)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d" podUID="538b59c0-d6e0-4926-be5c-a831cfda6b74"
Jul 17 13:15:44.773070 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:44.773041    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d"]
Jul 17 13:15:44.893365 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:44.893344    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d"
Jul 17 13:15:44.987112 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:44.987079    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538b59c0-d6e0-4926-be5c-a831cfda6b74-maas-api-service-ca\") pod \"538b59c0-d6e0-4926-be5c-a831cfda6b74\" (UID: \"538b59c0-d6e0-4926-be5c-a831cfda6b74\") "
Jul 17 13:15:44.987394 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:44.987375    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/538b59c0-d6e0-4926-be5c-a831cfda6b74-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "538b59c0-d6e0-4926-be5c-a831cfda6b74" (UID: "538b59c0-d6e0-4926-be5c-a831cfda6b74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:15:45.087868 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:45.087809    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/538b59c0-d6e0-4926-be5c-a831cfda6b74-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:15:45.734210 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:45.734181    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d"
Jul 17 13:15:45.734210 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:45.734189    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d" event={"ID":"538b59c0-d6e0-4926-be5c-a831cfda6b74","Type":"ContainerDied","Data":"9636172b652368ade05a818c871a96f379a5811db5692fd669f71496276f7311"}
Jul 17 13:15:45.734429 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:45.734257    2569 scope.go:117] "RemoveContainer" containerID="b976a3b11145b51fe40f71ca6e8c44a146a04e81b986bbd1f18f491b0d745f35"
Jul 17 13:15:45.758007 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:45.757976    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d"]
Jul 17 13:15:45.763050 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:45.763029    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-5dk5d"]
Jul 17 13:15:47.633904 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:47.633876    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="538b59c0-d6e0-4926-be5c-a831cfda6b74" path="/var/lib/kubelet/pods/538b59c0-d6e0-4926-be5c-a831cfda6b74/volumes"
Jul 17 13:15:57.629298 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:15:57.629196    2569 scope.go:117] "RemoveContainer" containerID="8bbce74974119b0104ff37782f24cebc82f73263db21b95299f5ee3a04cca985"
Jul 17 13:15:57.629829 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:15:57.629447    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:16:11.628371 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:11.628337    2569 scope.go:117] "RemoveContainer" containerID="8bbce74974119b0104ff37782f24cebc82f73263db21b95299f5ee3a04cca985"
Jul 17 13:16:11.628875 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:16:11.628522    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:16:12.016354 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016320    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l"]
Jul 17 13:16:12.016681 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016667    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="538b59c0-d6e0-4926-be5c-a831cfda6b74" containerName="revoke-keys"
Jul 17 13:16:12.016745 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016682    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="538b59c0-d6e0-4926-be5c-a831cfda6b74" containerName="revoke-keys"
Jul 17 13:16:12.016745 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016700    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="802eba4e-9187-425a-a189-3d5403ad7d6d" containerName="revoke-keys"
Jul 17 13:16:12.016745 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016705    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="802eba4e-9187-425a-a189-3d5403ad7d6d" containerName="revoke-keys"
Jul 17 13:16:12.016745 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016721    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="802eba4e-9187-425a-a189-3d5403ad7d6d" containerName="revoke-keys"
Jul 17 13:16:12.016745 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016727    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="802eba4e-9187-425a-a189-3d5403ad7d6d" containerName="revoke-keys"
Jul 17 13:16:12.016745 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016735    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cd2595b-e2a9-4277-91f2-e99519bca86e" containerName="revoke-keys"
Jul 17 13:16:12.016745 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016742    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cd2595b-e2a9-4277-91f2-e99519bca86e" containerName="revoke-keys"
Jul 17 13:16:12.017034 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016772    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="538b59c0-d6e0-4926-be5c-a831cfda6b74" containerName="revoke-keys"
Jul 17 13:16:12.017034 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016778    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="538b59c0-d6e0-4926-be5c-a831cfda6b74" containerName="revoke-keys"
Jul 17 13:16:12.017034 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016784    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="802eba4e-9187-425a-a189-3d5403ad7d6d" containerName="revoke-keys"
Jul 17 13:16:12.017034 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016791    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="802eba4e-9187-425a-a189-3d5403ad7d6d" containerName="revoke-keys"
Jul 17 13:16:12.017034 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016795    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cd2595b-e2a9-4277-91f2-e99519bca86e" containerName="revoke-keys"
Jul 17 13:16:12.017034 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016801    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cd2595b-e2a9-4277-91f2-e99519bca86e" containerName="revoke-keys"
Jul 17 13:16:12.017034 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016810    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cd2595b-e2a9-4277-91f2-e99519bca86e" containerName="revoke-keys"
Jul 17 13:16:12.017034 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016815    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cd2595b-e2a9-4277-91f2-e99519bca86e" containerName="revoke-keys"
Jul 17 13:16:12.017034 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016879    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="802eba4e-9187-425a-a189-3d5403ad7d6d" containerName="revoke-keys"
Jul 17 13:16:12.017034 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016887    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="538b59c0-d6e0-4926-be5c-a831cfda6b74" containerName="revoke-keys"
Jul 17 13:16:12.017034 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016894    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cd2595b-e2a9-4277-91f2-e99519bca86e" containerName="revoke-keys"
Jul 17 13:16:12.017034 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016899    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="538b59c0-d6e0-4926-be5c-a831cfda6b74" containerName="revoke-keys"
Jul 17 13:16:12.017034 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016907    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="538b59c0-d6e0-4926-be5c-a831cfda6b74" containerName="revoke-keys"
Jul 17 13:16:12.017034 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016913    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cd2595b-e2a9-4277-91f2-e99519bca86e" containerName="revoke-keys"
Jul 17 13:16:12.017034 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016919    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cd2595b-e2a9-4277-91f2-e99519bca86e" containerName="revoke-keys"
Jul 17 13:16:12.017034 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.016924    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="802eba4e-9187-425a-a189-3d5403ad7d6d" containerName="revoke-keys"
Jul 17 13:16:12.021650 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.021618    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l"
Jul 17 13:16:12.024876 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.024848    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:16:12.028888 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.028732    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l"]
Jul 17 13:16:12.086864 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.086837    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0305ae-76bb-4996-9b71-395f08f1f57e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l\" (UID: \"bc0305ae-76bb-4996-9b71-395f08f1f57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l"
Jul 17 13:16:12.187430 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.187399    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0305ae-76bb-4996-9b71-395f08f1f57e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l\" (UID: \"bc0305ae-76bb-4996-9b71-395f08f1f57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l"
Jul 17 13:16:12.187969 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.187953    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0305ae-76bb-4996-9b71-395f08f1f57e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l\" (UID: \"bc0305ae-76bb-4996-9b71-395f08f1f57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l"
Jul 17 13:16:12.334188 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.334129    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l"
Jul 17 13:16:12.480526 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:16:12.480494    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0305ae_76bb_4996_9b71_395f08f1f57e.slice/crio-2936e956db1466a25350e3aea41aba3d5533b1aece2e9d14c7ac0d74ec7ff1cd WatchSource:0}: Error finding container 2936e956db1466a25350e3aea41aba3d5533b1aece2e9d14c7ac0d74ec7ff1cd: Status 404 returned error can't find the container with id 2936e956db1466a25350e3aea41aba3d5533b1aece2e9d14c7ac0d74ec7ff1cd
Jul 17 13:16:12.480741 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.480720    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l"]
Jul 17 13:16:12.872978 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.872946    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l" event={"ID":"bc0305ae-76bb-4996-9b71-395f08f1f57e","Type":"ContainerStarted","Data":"15938aef69ce40f40a7d193048a2357056931f2b4d378c5a36516775d17d6f37"}
Jul 17 13:16:12.873319 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.872998    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l" event={"ID":"bc0305ae-76bb-4996-9b71-395f08f1f57e","Type":"ContainerStarted","Data":"2936e956db1466a25350e3aea41aba3d5533b1aece2e9d14c7ac0d74ec7ff1cd"}
Jul 17 13:16:12.891159 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:12.891084    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l" podStartSLOduration=1.891068795 podStartE2EDuration="1.891068795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:16:12.890455637 +0000 UTC m=+3247.894502519" watchObservedRunningTime="2026-07-17 13:16:12.891068795 +0000 UTC m=+3247.895115677"
Jul 17 13:16:13.878872 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:13.878835    2569 generic.go:356] "Generic (PLEG): container finished" podID="bc0305ae-76bb-4996-9b71-395f08f1f57e" containerID="15938aef69ce40f40a7d193048a2357056931f2b4d378c5a36516775d17d6f37" exitCode=6
Jul 17 13:16:13.879349 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:13.878924    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l" event={"ID":"bc0305ae-76bb-4996-9b71-395f08f1f57e","Type":"ContainerDied","Data":"15938aef69ce40f40a7d193048a2357056931f2b4d378c5a36516775d17d6f37"}
Jul 17 13:16:13.879349 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:13.879118    2569 scope.go:117] "RemoveContainer" containerID="15938aef69ce40f40a7d193048a2357056931f2b4d378c5a36516775d17d6f37"
Jul 17 13:16:14.885934 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:14.885905    2569 generic.go:356] "Generic (PLEG): container finished" podID="bc0305ae-76bb-4996-9b71-395f08f1f57e" containerID="b4d4d3651d7518ca8a76e5cb35e8fd4f8e67405130d75ecc05906b7bc122d1b4" exitCode=6
Jul 17 13:16:14.886415 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:14.885999    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l" event={"ID":"bc0305ae-76bb-4996-9b71-395f08f1f57e","Type":"ContainerDied","Data":"b4d4d3651d7518ca8a76e5cb35e8fd4f8e67405130d75ecc05906b7bc122d1b4"}
Jul 17 13:16:14.886415 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:14.886065    2569 scope.go:117] "RemoveContainer" containerID="15938aef69ce40f40a7d193048a2357056931f2b4d378c5a36516775d17d6f37"
Jul 17 13:16:14.886415 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:14.886234    2569 scope.go:117] "RemoveContainer" containerID="b4d4d3651d7518ca8a76e5cb35e8fd4f8e67405130d75ecc05906b7bc122d1b4"
Jul 17 13:16:14.886574 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:16:14.886485    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l_odh-ai-gateway-infra(bc0305ae-76bb-4996-9b71-395f08f1f57e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l" podUID="bc0305ae-76bb-4996-9b71-395f08f1f57e"
Jul 17 13:16:15.892712 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:15.892678    2569 scope.go:117] "RemoveContainer" containerID="b4d4d3651d7518ca8a76e5cb35e8fd4f8e67405130d75ecc05906b7bc122d1b4"
Jul 17 13:16:15.893171 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:16:15.892932    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l_odh-ai-gateway-infra(bc0305ae-76bb-4996-9b71-395f08f1f57e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l" podUID="bc0305ae-76bb-4996-9b71-395f08f1f57e"
Jul 17 13:16:18.354267 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:18.354206    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv"]
Jul 17 13:16:18.354680 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:18.354626    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="538b59c0-d6e0-4926-be5c-a831cfda6b74" containerName="revoke-keys"
Jul 17 13:16:18.354680 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:18.354639    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="538b59c0-d6e0-4926-be5c-a831cfda6b74" containerName="revoke-keys"
Jul 17 13:16:18.354807 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:18.354701    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="802eba4e-9187-425a-a189-3d5403ad7d6d" containerName="revoke-keys"
Jul 17 13:16:18.357727 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:18.357706    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv"
Jul 17 13:16:18.363630 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:18.363599    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv"]
Jul 17 13:16:18.438980 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:18.438954    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8c0acb-f971-4bbc-8132-e3b5da2f4970-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv\" (UID: \"ff8c0acb-f971-4bbc-8132-e3b5da2f4970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv"
Jul 17 13:16:18.539387 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:18.539355    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8c0acb-f971-4bbc-8132-e3b5da2f4970-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv\" (UID: \"ff8c0acb-f971-4bbc-8132-e3b5da2f4970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv"
Jul 17 13:16:18.539959 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:18.539942    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8c0acb-f971-4bbc-8132-e3b5da2f4970-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv\" (UID: \"ff8c0acb-f971-4bbc-8132-e3b5da2f4970\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv"
Jul 17 13:16:18.671479 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:18.671417    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv"
Jul 17 13:16:18.806570 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:18.806540    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv"]
Jul 17 13:16:18.810536 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:16:18.810501    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8c0acb_f971_4bbc_8132_e3b5da2f4970.slice/crio-739b266cb4b637be792b68f1b599bebdab6ca9d94f696f4ba24de4f2eb172f67 WatchSource:0}: Error finding container 739b266cb4b637be792b68f1b599bebdab6ca9d94f696f4ba24de4f2eb172f67: Status 404 returned error can't find the container with id 739b266cb4b637be792b68f1b599bebdab6ca9d94f696f4ba24de4f2eb172f67
Jul 17 13:16:18.911274 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:18.911252    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv" event={"ID":"ff8c0acb-f971-4bbc-8132-e3b5da2f4970","Type":"ContainerStarted","Data":"739b266cb4b637be792b68f1b599bebdab6ca9d94f696f4ba24de4f2eb172f67"}
Jul 17 13:16:19.918336 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:19.918303    2569 generic.go:356] "Generic (PLEG): container finished" podID="ff8c0acb-f971-4bbc-8132-e3b5da2f4970" containerID="1a049bc2887c35eb5f4a10301d8c0bd72e3047393c2525bf1f9232847647a025" exitCode=7
Jul 17 13:16:19.918714 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:19.918396    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv" event={"ID":"ff8c0acb-f971-4bbc-8132-e3b5da2f4970","Type":"ContainerDied","Data":"1a049bc2887c35eb5f4a10301d8c0bd72e3047393c2525bf1f9232847647a025"}
Jul 17 13:16:19.918714 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:19.918538    2569 scope.go:117] "RemoveContainer" containerID="1a049bc2887c35eb5f4a10301d8c0bd72e3047393c2525bf1f9232847647a025"
Jul 17 13:16:20.874346 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:20.874282    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x"]
Jul 17 13:16:20.877583 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:20.877565    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x"
Jul 17 13:16:20.887410 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:20.887382    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x"]
Jul 17 13:16:20.925847 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:20.925826    2569 generic.go:356] "Generic (PLEG): container finished" podID="ff8c0acb-f971-4bbc-8132-e3b5da2f4970" containerID="ec80b05239134e2df146dbdbf44ac8e98eeb532a88f3d3e1231d10e50b0c6273" exitCode=7
Jul 17 13:16:20.926187 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:20.925918    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv" event={"ID":"ff8c0acb-f971-4bbc-8132-e3b5da2f4970","Type":"ContainerDied","Data":"ec80b05239134e2df146dbdbf44ac8e98eeb532a88f3d3e1231d10e50b0c6273"}
Jul 17 13:16:20.926187 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:20.925976    2569 scope.go:117] "RemoveContainer" containerID="1a049bc2887c35eb5f4a10301d8c0bd72e3047393c2525bf1f9232847647a025"
Jul 17 13:16:20.926187 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:20.926105    2569 scope.go:117] "RemoveContainer" containerID="ec80b05239134e2df146dbdbf44ac8e98eeb532a88f3d3e1231d10e50b0c6273"
Jul 17 13:16:20.926351 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:16:20.926321    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv_odh-ai-gateway-infra(ff8c0acb-f971-4bbc-8132-e3b5da2f4970)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv" podUID="ff8c0acb-f971-4bbc-8132-e3b5da2f4970"
Jul 17 13:16:20.957973 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:20.957953    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f6eb5d4-ed3c-40ec-9267-72e948722b3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x\" (UID: \"6f6eb5d4-ed3c-40ec-9267-72e948722b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x"
Jul 17 13:16:21.059269 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:21.059247    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f6eb5d4-ed3c-40ec-9267-72e948722b3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x\" (UID: \"6f6eb5d4-ed3c-40ec-9267-72e948722b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x"
Jul 17 13:16:21.059803 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:21.059785    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f6eb5d4-ed3c-40ec-9267-72e948722b3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x\" (UID: \"6f6eb5d4-ed3c-40ec-9267-72e948722b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x"
Jul 17 13:16:21.189736 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:21.189680    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x"
Jul 17 13:16:21.317010 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:21.316984    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x"]
Jul 17 13:16:21.318718 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:16:21.318688    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6eb5d4_ed3c_40ec_9267_72e948722b3b.slice/crio-c5f9799137fb8a129b17ef05298f5d865de06d05baa7c02e0e67e46b14492263 WatchSource:0}: Error finding container c5f9799137fb8a129b17ef05298f5d865de06d05baa7c02e0e67e46b14492263: Status 404 returned error can't find the container with id c5f9799137fb8a129b17ef05298f5d865de06d05baa7c02e0e67e46b14492263
Jul 17 13:16:21.933356 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:21.933332    2569 scope.go:117] "RemoveContainer" containerID="ec80b05239134e2df146dbdbf44ac8e98eeb532a88f3d3e1231d10e50b0c6273"
Jul 17 13:16:21.933775 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:16:21.933570    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv_odh-ai-gateway-infra(ff8c0acb-f971-4bbc-8132-e3b5da2f4970)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv" podUID="ff8c0acb-f971-4bbc-8132-e3b5da2f4970"
Jul 17 13:16:21.935027 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:21.934993    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x" event={"ID":"6f6eb5d4-ed3c-40ec-9267-72e948722b3b","Type":"ContainerStarted","Data":"6984433f638a2a572ab56bd01ce087ac49fea6c8ba1751849c664cd212a5f144"}
Jul 17 13:16:21.935121 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:21.935061    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x" event={"ID":"6f6eb5d4-ed3c-40ec-9267-72e948722b3b","Type":"ContainerStarted","Data":"c5f9799137fb8a129b17ef05298f5d865de06d05baa7c02e0e67e46b14492263"}
Jul 17 13:16:21.967310 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:21.967268    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x" podStartSLOduration=1.967252869 podStartE2EDuration="1.967252869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:16:21.965244028 +0000 UTC m=+3256.969290903" watchObservedRunningTime="2026-07-17 13:16:21.967252869 +0000 UTC m=+3256.971299751"
Jul 17 13:16:22.941100 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:22.941069    2569 generic.go:356] "Generic (PLEG): container finished" podID="6f6eb5d4-ed3c-40ec-9267-72e948722b3b" containerID="6984433f638a2a572ab56bd01ce087ac49fea6c8ba1751849c664cd212a5f144" exitCode=6
Jul 17 13:16:22.941443 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:22.941154    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x" event={"ID":"6f6eb5d4-ed3c-40ec-9267-72e948722b3b","Type":"ContainerDied","Data":"6984433f638a2a572ab56bd01ce087ac49fea6c8ba1751849c664cd212a5f144"}
Jul 17 13:16:22.941443 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:22.941321    2569 scope.go:117] "RemoveContainer" containerID="6984433f638a2a572ab56bd01ce087ac49fea6c8ba1751849c664cd212a5f144"
Jul 17 13:16:23.947643 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:23.947573    2569 generic.go:356] "Generic (PLEG): container finished" podID="6f6eb5d4-ed3c-40ec-9267-72e948722b3b" containerID="8fdc09e9a68f4b355b67dfe81ea74d43a43dfe486d174a8d6f324350c66d028a" exitCode=6
Jul 17 13:16:23.948074 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:23.947659    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x" event={"ID":"6f6eb5d4-ed3c-40ec-9267-72e948722b3b","Type":"ContainerDied","Data":"8fdc09e9a68f4b355b67dfe81ea74d43a43dfe486d174a8d6f324350c66d028a"}
Jul 17 13:16:23.948074 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:23.947725    2569 scope.go:117] "RemoveContainer" containerID="6984433f638a2a572ab56bd01ce087ac49fea6c8ba1751849c664cd212a5f144"
Jul 17 13:16:23.948074 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:23.947931    2569 scope.go:117] "RemoveContainer" containerID="8fdc09e9a68f4b355b67dfe81ea74d43a43dfe486d174a8d6f324350c66d028a"
Jul 17 13:16:23.948242 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:16:23.948132    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x_odh-ai-gateway-infra(6f6eb5d4-ed3c-40ec-9267-72e948722b3b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x" podUID="6f6eb5d4-ed3c-40ec-9267-72e948722b3b"
Jul 17 13:16:24.955459 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:24.955423    2569 scope.go:117] "RemoveContainer" containerID="8fdc09e9a68f4b355b67dfe81ea74d43a43dfe486d174a8d6f324350c66d028a"
Jul 17 13:16:24.956038 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:16:24.955657    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x_odh-ai-gateway-infra(6f6eb5d4-ed3c-40ec-9267-72e948722b3b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x" podUID="6f6eb5d4-ed3c-40ec-9267-72e948722b3b"
Jul 17 13:16:25.631052 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:25.631026    2569 scope.go:117] "RemoveContainer" containerID="8bbce74974119b0104ff37782f24cebc82f73263db21b95299f5ee3a04cca985"
Jul 17 13:16:26.628649 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:26.628629    2569 scope.go:117] "RemoveContainer" containerID="b4d4d3651d7518ca8a76e5cb35e8fd4f8e67405130d75ecc05906b7bc122d1b4"
Jul 17 13:16:26.965580 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:26.965554    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l" event={"ID":"bc0305ae-76bb-4996-9b71-395f08f1f57e","Type":"ContainerStarted","Data":"5f6d017873761fb9f69c01098f5ca238af968dec8448b8e6a4c28e85f88f752c"}
Jul 17 13:16:26.967344 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:26.967323    2569 generic.go:356] "Generic (PLEG): container finished" podID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerID="2e490d49561a392018b3eb0c6313ce11bf343f712229244ff8c33c8b44d22f9e" exitCode=1
Jul 17 13:16:26.967438 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:26.967393    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" event={"ID":"2c1b598d-1c76-4afd-8c32-c50b57a701c5","Type":"ContainerDied","Data":"2e490d49561a392018b3eb0c6313ce11bf343f712229244ff8c33c8b44d22f9e"}
Jul 17 13:16:26.967480 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:26.967445    2569 scope.go:117] "RemoveContainer" containerID="8bbce74974119b0104ff37782f24cebc82f73263db21b95299f5ee3a04cca985"
Jul 17 13:16:26.967700 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:26.967685    2569 scope.go:117] "RemoveContainer" containerID="2e490d49561a392018b3eb0c6313ce11bf343f712229244ff8c33c8b44d22f9e"
Jul 17 13:16:26.967882 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:16:26.967864    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:16:27.975117 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:27.975080    2569 generic.go:356] "Generic (PLEG): container finished" podID="bc0305ae-76bb-4996-9b71-395f08f1f57e" containerID="5f6d017873761fb9f69c01098f5ca238af968dec8448b8e6a4c28e85f88f752c" exitCode=6
Jul 17 13:16:27.975569 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:27.975168    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l" event={"ID":"bc0305ae-76bb-4996-9b71-395f08f1f57e","Type":"ContainerDied","Data":"5f6d017873761fb9f69c01098f5ca238af968dec8448b8e6a4c28e85f88f752c"}
Jul 17 13:16:27.975569 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:27.975252    2569 scope.go:117] "RemoveContainer" containerID="b4d4d3651d7518ca8a76e5cb35e8fd4f8e67405130d75ecc05906b7bc122d1b4"
Jul 17 13:16:27.975569 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:27.975552    2569 scope.go:117] "RemoveContainer" containerID="5f6d017873761fb9f69c01098f5ca238af968dec8448b8e6a4c28e85f88f752c"
Jul 17 13:16:27.975799 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:16:27.975772    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l_odh-ai-gateway-infra(bc0305ae-76bb-4996-9b71-395f08f1f57e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l" podUID="bc0305ae-76bb-4996-9b71-395f08f1f57e"
Jul 17 13:16:27.990908 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:27.990883    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l"]
Jul 17 13:16:29.124941 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:29.124918    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l"
Jul 17 13:16:29.221301 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:29.221274    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0305ae-76bb-4996-9b71-395f08f1f57e-maas-api-service-ca\") pod \"bc0305ae-76bb-4996-9b71-395f08f1f57e\" (UID: \"bc0305ae-76bb-4996-9b71-395f08f1f57e\") "
Jul 17 13:16:29.221632 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:29.221614    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc0305ae-76bb-4996-9b71-395f08f1f57e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc0305ae-76bb-4996-9b71-395f08f1f57e" (UID: "bc0305ae-76bb-4996-9b71-395f08f1f57e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:16:29.232247 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:29.232224    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:16:29.232536 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:29.232524    2569 scope.go:117] "RemoveContainer" containerID="2e490d49561a392018b3eb0c6313ce11bf343f712229244ff8c33c8b44d22f9e"
Jul 17 13:16:29.232685 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:16:29.232668    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:16:29.322054 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:29.321994    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc0305ae-76bb-4996-9b71-395f08f1f57e-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:16:29.990332 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:29.990305    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l"
Jul 17 13:16:29.990470 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:29.990304    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l" event={"ID":"bc0305ae-76bb-4996-9b71-395f08f1f57e","Type":"ContainerDied","Data":"2936e956db1466a25350e3aea41aba3d5533b1aece2e9d14c7ac0d74ec7ff1cd"}
Jul 17 13:16:29.990470 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:29.990441    2569 scope.go:117] "RemoveContainer" containerID="5f6d017873761fb9f69c01098f5ca238af968dec8448b8e6a4c28e85f88f752c"
Jul 17 13:16:30.010687 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:30.010654    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l"]
Jul 17 13:16:30.012719 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:30.012698    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lvp5l"]
Jul 17 13:16:31.634726 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:31.634688    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc0305ae-76bb-4996-9b71-395f08f1f57e" path="/var/lib/kubelet/pods/bc0305ae-76bb-4996-9b71-395f08f1f57e/volumes"
Jul 17 13:16:34.172181 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:34.172139    2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:16:34.172583 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:34.172542    2569 scope.go:117] "RemoveContainer" containerID="2e490d49561a392018b3eb0c6313ce11bf343f712229244ff8c33c8b44d22f9e"
Jul 17 13:16:34.172734 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:16:34.172711    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-661ee8a2-594dbcc585-js747_odh-ai-gateway-infra(2c1b598d-1c76-4afd-8c32-c50b57a701c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5"
Jul 17 13:16:34.628910 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:34.628884    2569 scope.go:117] "RemoveContainer" containerID="ec80b05239134e2df146dbdbf44ac8e98eeb532a88f3d3e1231d10e50b0c6273"
Jul 17 13:16:35.017661 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:35.017636    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv" event={"ID":"ff8c0acb-f971-4bbc-8132-e3b5da2f4970","Type":"ContainerStarted","Data":"e85dcacfa9e592e9a6aabd0eb39b5f0507499af33aaf01d0b734cfadfdf20cb4"}
Jul 17 13:16:35.035195 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:35.035153    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv" podStartSLOduration=17.035136165 podStartE2EDuration="17.035136165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:16:35.034805916 +0000 UTC m=+3270.038852798" watchObservedRunningTime="2026-07-17 13:16:35.035136165 +0000 UTC m=+3270.039183048"
Jul 17 13:16:36.023575 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:36.023544    2569 generic.go:356] "Generic (PLEG): container finished" podID="ff8c0acb-f971-4bbc-8132-e3b5da2f4970" containerID="e85dcacfa9e592e9a6aabd0eb39b5f0507499af33aaf01d0b734cfadfdf20cb4" exitCode=7
Jul 17 13:16:36.023990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:36.023588    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv" event={"ID":"ff8c0acb-f971-4bbc-8132-e3b5da2f4970","Type":"ContainerDied","Data":"e85dcacfa9e592e9a6aabd0eb39b5f0507499af33aaf01d0b734cfadfdf20cb4"}
Jul 17 13:16:36.023990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:36.023636    2569 scope.go:117] "RemoveContainer" containerID="ec80b05239134e2df146dbdbf44ac8e98eeb532a88f3d3e1231d10e50b0c6273"
Jul 17 13:16:36.023990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:36.023874    2569 scope.go:117] "RemoveContainer" containerID="e85dcacfa9e592e9a6aabd0eb39b5f0507499af33aaf01d0b734cfadfdf20cb4"
Jul 17 13:16:36.024096 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:16:36.024058    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv_odh-ai-gateway-infra(ff8c0acb-f971-4bbc-8132-e3b5da2f4970)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv" podUID="ff8c0acb-f971-4bbc-8132-e3b5da2f4970"
Jul 17 13:16:36.051789 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:36.051763    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv"]
Jul 17 13:16:36.628028 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:36.628006    2569 scope.go:117] "RemoveContainer" containerID="8fdc09e9a68f4b355b67dfe81ea74d43a43dfe486d174a8d6f324350c66d028a"
Jul 17 13:16:37.031881 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:37.031854    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x" event={"ID":"6f6eb5d4-ed3c-40ec-9267-72e948722b3b","Type":"ContainerStarted","Data":"42bb71c8a3ff6ab04628c7d7ced0829ea6ce70030061755cc8980ade3cf4602e"}
Jul 17 13:16:37.158084 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:37.158061    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv"
Jul 17 13:16:37.283694 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:37.283666    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8c0acb-f971-4bbc-8132-e3b5da2f4970-maas-api-service-ca\") pod \"ff8c0acb-f971-4bbc-8132-e3b5da2f4970\" (UID: \"ff8c0acb-f971-4bbc-8132-e3b5da2f4970\") "
Jul 17 13:16:37.284041 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:37.284022    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff8c0acb-f971-4bbc-8132-e3b5da2f4970-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff8c0acb-f971-4bbc-8132-e3b5da2f4970" (UID: "ff8c0acb-f971-4bbc-8132-e3b5da2f4970"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:16:37.385197 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:37.385173    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff8c0acb-f971-4bbc-8132-e3b5da2f4970-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:16:38.037090 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:38.037066    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv"
Jul 17 13:16:38.037508 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:38.037065    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv" event={"ID":"ff8c0acb-f971-4bbc-8132-e3b5da2f4970","Type":"ContainerDied","Data":"739b266cb4b637be792b68f1b599bebdab6ca9d94f696f4ba24de4f2eb172f67"}
Jul 17 13:16:38.037508 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:38.037201    2569 scope.go:117] "RemoveContainer" containerID="e85dcacfa9e592e9a6aabd0eb39b5f0507499af33aaf01d0b734cfadfdf20cb4"
Jul 17 13:16:38.039712 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:38.039686    2569 generic.go:356] "Generic (PLEG): container finished" podID="6f6eb5d4-ed3c-40ec-9267-72e948722b3b" containerID="42bb71c8a3ff6ab04628c7d7ced0829ea6ce70030061755cc8980ade3cf4602e" exitCode=6
Jul 17 13:16:38.039828 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:38.039784    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x" event={"ID":"6f6eb5d4-ed3c-40ec-9267-72e948722b3b","Type":"ContainerDied","Data":"42bb71c8a3ff6ab04628c7d7ced0829ea6ce70030061755cc8980ade3cf4602e"}
Jul 17 13:16:38.040132 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:38.040114    2569 scope.go:117] "RemoveContainer" containerID="42bb71c8a3ff6ab04628c7d7ced0829ea6ce70030061755cc8980ade3cf4602e"
Jul 17 13:16:38.040347 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:16:38.040327    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x_odh-ai-gateway-infra(6f6eb5d4-ed3c-40ec-9267-72e948722b3b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x" podUID="6f6eb5d4-ed3c-40ec-9267-72e948722b3b"
Jul 17 13:16:38.045920 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:38.045904    2569 scope.go:117] "RemoveContainer" containerID="8fdc09e9a68f4b355b67dfe81ea74d43a43dfe486d174a8d6f324350c66d028a"
Jul 17 13:16:38.058137 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:38.058113    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv"]
Jul 17 13:16:38.061324 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:38.061300    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-n4pqv"]
Jul 17 13:16:38.077835 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:38.077814    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x"]
Jul 17 13:16:39.178395 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:39.178370    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x"
Jul 17 13:16:39.301127 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:39.301055    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f6eb5d4-ed3c-40ec-9267-72e948722b3b-maas-api-service-ca\") pod \"6f6eb5d4-ed3c-40ec-9267-72e948722b3b\" (UID: \"6f6eb5d4-ed3c-40ec-9267-72e948722b3b\") "
Jul 17 13:16:39.301371 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:39.301350    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f6eb5d4-ed3c-40ec-9267-72e948722b3b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f6eb5d4-ed3c-40ec-9267-72e948722b3b" (UID: "6f6eb5d4-ed3c-40ec-9267-72e948722b3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:16:39.401629 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:39.401602    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f6eb5d4-ed3c-40ec-9267-72e948722b3b-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:16:39.640085 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:39.640019    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff8c0acb-f971-4bbc-8132-e3b5da2f4970" path="/var/lib/kubelet/pods/ff8c0acb-f971-4bbc-8132-e3b5da2f4970/volumes"
Jul 17 13:16:40.053723 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:40.053699    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x"
Jul 17 13:16:40.053884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:40.053699    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x" event={"ID":"6f6eb5d4-ed3c-40ec-9267-72e948722b3b","Type":"ContainerDied","Data":"c5f9799137fb8a129b17ef05298f5d865de06d05baa7c02e0e67e46b14492263"}
Jul 17 13:16:40.053884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:40.053852    2569 scope.go:117] "RemoveContainer" containerID="42bb71c8a3ff6ab04628c7d7ced0829ea6ce70030061755cc8980ade3cf4602e"
Jul 17 13:16:40.070129 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:40.070102    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x"]
Jul 17 13:16:40.074469 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:40.074444    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-fsb8x"]
Jul 17 13:16:41.173736 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:41.173705    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:16:41.635313 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:41.635279    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f6eb5d4-ed3c-40ec-9267-72e948722b3b" path="/var/lib/kubelet/pods/6f6eb5d4-ed3c-40ec-9267-72e948722b3b/volumes"
Jul 17 13:16:44.430765 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:44.430717    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"]
Jul 17 13:16:44.562742 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:44.562716    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:16:44.646921 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:44.646894    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c1b598d-1c76-4afd-8c32-c50b57a701c5-maas-api-tls\") pod \"2c1b598d-1c76-4afd-8c32-c50b57a701c5\" (UID: \"2c1b598d-1c76-4afd-8c32-c50b57a701c5\") "
Jul 17 13:16:44.647046 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:44.646930    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1b598d-1c76-4afd-8c32-c50b57a701c5-openshift-service-ca\") pod \"2c1b598d-1c76-4afd-8c32-c50b57a701c5\" (UID: \"2c1b598d-1c76-4afd-8c32-c50b57a701c5\") "
Jul 17 13:16:44.647046 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:44.646988    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fd9w\" (UniqueName: \"kubernetes.io/projected/2c1b598d-1c76-4afd-8c32-c50b57a701c5-kube-api-access-9fd9w\") pod \"2c1b598d-1c76-4afd-8c32-c50b57a701c5\" (UID: \"2c1b598d-1c76-4afd-8c32-c50b57a701c5\") "
Jul 17 13:16:44.647339 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:44.647316    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1b598d-1c76-4afd-8c32-c50b57a701c5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c1b598d-1c76-4afd-8c32-c50b57a701c5" (UID: "2c1b598d-1c76-4afd-8c32-c50b57a701c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:16:44.649014 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:44.648993    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1b598d-1c76-4afd-8c32-c50b57a701c5-kube-api-access-9fd9w" (OuterVolumeSpecName: "kube-api-access-9fd9w") pod "2c1b598d-1c76-4afd-8c32-c50b57a701c5" (UID: "2c1b598d-1c76-4afd-8c32-c50b57a701c5"). InnerVolumeSpecName "kube-api-access-9fd9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:16:44.649115 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:44.649018    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c1b598d-1c76-4afd-8c32-c50b57a701c5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c1b598d-1c76-4afd-8c32-c50b57a701c5" (UID: "2c1b598d-1c76-4afd-8c32-c50b57a701c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:16:44.748337 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:44.748316    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fd9w\" (UniqueName: \"kubernetes.io/projected/2c1b598d-1c76-4afd-8c32-c50b57a701c5-kube-api-access-9fd9w\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:16:44.748428 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:44.748338    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c1b598d-1c76-4afd-8c32-c50b57a701c5-maas-api-tls\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:16:44.748428 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:44.748348    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1b598d-1c76-4afd-8c32-c50b57a701c5-openshift-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:16:45.080101 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:45.080048    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"
Jul 17 13:16:45.080101 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:45.080059    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747" event={"ID":"2c1b598d-1c76-4afd-8c32-c50b57a701c5","Type":"ContainerDied","Data":"df33c8d58ce20864f5a03b6bc4ea168e3145547487126c80fb80f5cf37ef01d8"}
Jul 17 13:16:45.080248 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:45.080115    2569 scope.go:117] "RemoveContainer" containerID="2e490d49561a392018b3eb0c6313ce11bf343f712229244ff8c33c8b44d22f9e"
Jul 17 13:16:45.101350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:45.101327    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"]
Jul 17 13:16:45.107698 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:45.107675    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-661ee8a2-594dbcc585-js747"]
Jul 17 13:16:45.633802 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:45.633765    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" path="/var/lib/kubelet/pods/2c1b598d-1c76-4afd-8c32-c50b57a701c5/volumes"
Jul 17 13:16:59.731334 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731296    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l"]
Jul 17 13:16:59.731842 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731731    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerName="maas-api"
Jul 17 13:16:59.731842 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731766    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerName="maas-api"
Jul 17 13:16:59.731842 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731778    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc0305ae-76bb-4996-9b71-395f08f1f57e" containerName="revoke-keys"
Jul 17 13:16:59.731842 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731787    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc0305ae-76bb-4996-9b71-395f08f1f57e" containerName="revoke-keys"
Jul 17 13:16:59.731842 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731797    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerName="maas-api"
Jul 17 13:16:59.731842 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731805    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerName="maas-api"
Jul 17 13:16:59.731842 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731814    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerName="maas-api"
Jul 17 13:16:59.731842 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731822    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerName="maas-api"
Jul 17 13:16:59.731842 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731831    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff8c0acb-f971-4bbc-8132-e3b5da2f4970" containerName="revoke-keys"
Jul 17 13:16:59.731842 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731839    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff8c0acb-f971-4bbc-8132-e3b5da2f4970" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731851    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc0305ae-76bb-4996-9b71-395f08f1f57e" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731859    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc0305ae-76bb-4996-9b71-395f08f1f57e" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731868    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f6eb5d4-ed3c-40ec-9267-72e948722b3b" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731876    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f6eb5d4-ed3c-40ec-9267-72e948722b3b" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731885    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f6eb5d4-ed3c-40ec-9267-72e948722b3b" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731893    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f6eb5d4-ed3c-40ec-9267-72e948722b3b" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731913    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff8c0acb-f971-4bbc-8132-e3b5da2f4970" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731921    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff8c0acb-f971-4bbc-8132-e3b5da2f4970" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731935    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerName="maas-api"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731944    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerName="maas-api"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731956    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f6eb5d4-ed3c-40ec-9267-72e948722b3b" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.731964    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f6eb5d4-ed3c-40ec-9267-72e948722b3b" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.732044    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc0305ae-76bb-4996-9b71-395f08f1f57e" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.732055    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f6eb5d4-ed3c-40ec-9267-72e948722b3b" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.732065    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f6eb5d4-ed3c-40ec-9267-72e948722b3b" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.732075    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc0305ae-76bb-4996-9b71-395f08f1f57e" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.732085    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff8c0acb-f971-4bbc-8132-e3b5da2f4970" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.732094    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff8c0acb-f971-4bbc-8132-e3b5da2f4970" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.732103    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerName="maas-api"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.732111    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc0305ae-76bb-4996-9b71-395f08f1f57e" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.732120    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff8c0acb-f971-4bbc-8132-e3b5da2f4970" containerName="revoke-keys"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.732129    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerName="maas-api"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.732140    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerName="maas-api"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.732148    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerName="maas-api"
Jul 17 13:16:59.732350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.732156    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerName="maas-api"
Jul 17 13:16:59.735344 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.735323    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l"
Jul 17 13:16:59.738053 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.738027    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:16:59.738168 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.738103    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:16:59.744645 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.744623    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l"]
Jul 17 13:16:59.859644 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.859619    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4d00a1-2462-4afb-b629-70bdecdda560-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l\" (UID: \"ed4d00a1-2462-4afb-b629-70bdecdda560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l"
Jul 17 13:16:59.960050 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.960028    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4d00a1-2462-4afb-b629-70bdecdda560-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l\" (UID: \"ed4d00a1-2462-4afb-b629-70bdecdda560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l"
Jul 17 13:16:59.960584 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:16:59.960567    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4d00a1-2462-4afb-b629-70bdecdda560-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l\" (UID: \"ed4d00a1-2462-4afb-b629-70bdecdda560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l"
Jul 17 13:17:00.046181 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:00.046122    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l"
Jul 17 13:17:00.174583 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:00.174552    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l"]
Jul 17 13:17:00.177963 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:17:00.177938    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4d00a1_2462_4afb_b629_70bdecdda560.slice/crio-46851b75f5354bc9ff95c19a105ddd90f51bc1e3dca0ebd975598de93deaec99 WatchSource:0}: Error finding container 46851b75f5354bc9ff95c19a105ddd90f51bc1e3dca0ebd975598de93deaec99: Status 404 returned error can't find the container with id 46851b75f5354bc9ff95c19a105ddd90f51bc1e3dca0ebd975598de93deaec99
Jul 17 13:17:01.164772 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:01.164681    2569 generic.go:356] "Generic (PLEG): container finished" podID="ed4d00a1-2462-4afb-b629-70bdecdda560" containerID="cc932547460cc8c904de8741c32647220b72822ab209ee8048c9c07459997486" exitCode=6
Jul 17 13:17:01.164772 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:01.164729    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l" event={"ID":"ed4d00a1-2462-4afb-b629-70bdecdda560","Type":"ContainerDied","Data":"cc932547460cc8c904de8741c32647220b72822ab209ee8048c9c07459997486"}
Jul 17 13:17:01.165147 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:01.164790    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l" event={"ID":"ed4d00a1-2462-4afb-b629-70bdecdda560","Type":"ContainerStarted","Data":"46851b75f5354bc9ff95c19a105ddd90f51bc1e3dca0ebd975598de93deaec99"}
Jul 17 13:17:01.165147 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:01.164974    2569 scope.go:117] "RemoveContainer" containerID="cc932547460cc8c904de8741c32647220b72822ab209ee8048c9c07459997486"
Jul 17 13:17:02.069990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.069960    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7"]
Jul 17 13:17:02.070273 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.070262    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerName="maas-api"
Jul 17 13:17:02.070318 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.070275    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerName="maas-api"
Jul 17 13:17:02.070318 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.070287    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc0305ae-76bb-4996-9b71-395f08f1f57e" containerName="revoke-keys"
Jul 17 13:17:02.070318 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.070293    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc0305ae-76bb-4996-9b71-395f08f1f57e" containerName="revoke-keys"
Jul 17 13:17:02.070318 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.070305    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff8c0acb-f971-4bbc-8132-e3b5da2f4970" containerName="revoke-keys"
Jul 17 13:17:02.070318 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.070311    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff8c0acb-f971-4bbc-8132-e3b5da2f4970" containerName="revoke-keys"
Jul 17 13:17:02.070471 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.070369    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f6eb5d4-ed3c-40ec-9267-72e948722b3b" containerName="revoke-keys"
Jul 17 13:17:02.070471 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.070380    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerName="maas-api"
Jul 17 13:17:02.073196 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.073178    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7"
Jul 17 13:17:02.083050 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.083027    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7"]
Jul 17 13:17:02.170429 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.170398    2569 generic.go:356] "Generic (PLEG): container finished" podID="ed4d00a1-2462-4afb-b629-70bdecdda560" containerID="a47e8bd5ab0b6d1cbb77581295bcc8a3d80f3ffa460c02aff5e7d529e759df7a" exitCode=6
Jul 17 13:17:02.170837 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.170465    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l" event={"ID":"ed4d00a1-2462-4afb-b629-70bdecdda560","Type":"ContainerDied","Data":"a47e8bd5ab0b6d1cbb77581295bcc8a3d80f3ffa460c02aff5e7d529e759df7a"}
Jul 17 13:17:02.170837 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.170514    2569 scope.go:117] "RemoveContainer" containerID="cc932547460cc8c904de8741c32647220b72822ab209ee8048c9c07459997486"
Jul 17 13:17:02.170837 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.170677    2569 scope.go:117] "RemoveContainer" containerID="a47e8bd5ab0b6d1cbb77581295bcc8a3d80f3ffa460c02aff5e7d529e759df7a"
Jul 17 13:17:02.171004 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:17:02.170901    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l_odh-ai-gateway-infra(ed4d00a1-2462-4afb-b629-70bdecdda560)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l" podUID="ed4d00a1-2462-4afb-b629-70bdecdda560"
Jul 17 13:17:02.173777 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.173758    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7\" (UID: \"fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7"
Jul 17 13:17:02.274435 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.274414    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7\" (UID: \"fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7"
Jul 17 13:17:02.274951 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.274934    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7\" (UID: \"fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7"
Jul 17 13:17:02.383727 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.383670    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7"
Jul 17 13:17:02.514979 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:02.514957    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7"]
Jul 17 13:17:02.516599 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:17:02.516572    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe07ce37_8dc6_498d_b0fa_9c67d3c3e7e3.slice/crio-e88484bdcac8080a827350b0d2fa452ca327e48464b51dd1e46a45ed834370e9 WatchSource:0}: Error finding container e88484bdcac8080a827350b0d2fa452ca327e48464b51dd1e46a45ed834370e9: Status 404 returned error can't find the container with id e88484bdcac8080a827350b0d2fa452ca327e48464b51dd1e46a45ed834370e9
Jul 17 13:17:03.177417 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:03.177392    2569 scope.go:117] "RemoveContainer" containerID="a47e8bd5ab0b6d1cbb77581295bcc8a3d80f3ffa460c02aff5e7d529e759df7a"
Jul 17 13:17:03.177741 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:17:03.177572    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l_odh-ai-gateway-infra(ed4d00a1-2462-4afb-b629-70bdecdda560)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l" podUID="ed4d00a1-2462-4afb-b629-70bdecdda560"
Jul 17 13:17:03.178696 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:03.178673    2569 generic.go:356] "Generic (PLEG): container finished" podID="fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3" containerID="516b459e3ac2e6debc74c3972d418bd743335d3fbf9840ba298e3c47a07599b4" exitCode=6
Jul 17 13:17:03.178800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:03.178736    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7" event={"ID":"fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3","Type":"ContainerDied","Data":"516b459e3ac2e6debc74c3972d418bd743335d3fbf9840ba298e3c47a07599b4"}
Jul 17 13:17:03.178860 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:03.178806    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7" event={"ID":"fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3","Type":"ContainerStarted","Data":"e88484bdcac8080a827350b0d2fa452ca327e48464b51dd1e46a45ed834370e9"}
Jul 17 13:17:03.178925 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:03.178910    2569 scope.go:117] "RemoveContainer" containerID="516b459e3ac2e6debc74c3972d418bd743335d3fbf9840ba298e3c47a07599b4"
Jul 17 13:17:04.185431 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:04.185402    2569 generic.go:356] "Generic (PLEG): container finished" podID="fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3" containerID="335ba0976bee2fa0919a5287be21beceb0800fe3bff93c47f003299c8c756cda" exitCode=6
Jul 17 13:17:04.185884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:04.185444    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7" event={"ID":"fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3","Type":"ContainerDied","Data":"335ba0976bee2fa0919a5287be21beceb0800fe3bff93c47f003299c8c756cda"}
Jul 17 13:17:04.185884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:04.185496    2569 scope.go:117] "RemoveContainer" containerID="516b459e3ac2e6debc74c3972d418bd743335d3fbf9840ba298e3c47a07599b4"
Jul 17 13:17:04.185884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:04.185714    2569 scope.go:117] "RemoveContainer" containerID="335ba0976bee2fa0919a5287be21beceb0800fe3bff93c47f003299c8c756cda"
Jul 17 13:17:04.186044 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:17:04.185940    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7_odh-ai-gateway-infra(fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7" podUID="fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3"
Jul 17 13:17:05.192568 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:05.192532    2569 scope.go:117] "RemoveContainer" containerID="335ba0976bee2fa0919a5287be21beceb0800fe3bff93c47f003299c8c756cda"
Jul 17 13:17:05.192991 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:17:05.192705    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7_odh-ai-gateway-infra(fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7" podUID="fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3"
Jul 17 13:17:08.392170 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:08.392129    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x"]
Jul 17 13:17:08.392643 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:08.392486    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerName="maas-api"
Jul 17 13:17:08.392643 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:08.392500    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1b598d-1c76-4afd-8c32-c50b57a701c5" containerName="maas-api"
Jul 17 13:17:08.396453 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:08.396433    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x"
Jul 17 13:17:08.402914 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:08.402855    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x"]
Jul 17 13:17:08.519176 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:08.519149    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ea86bd-7677-48fa-9dbe-481a08c1951d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x\" (UID: \"72ea86bd-7677-48fa-9dbe-481a08c1951d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x"
Jul 17 13:17:08.620116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:08.620093    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ea86bd-7677-48fa-9dbe-481a08c1951d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x\" (UID: \"72ea86bd-7677-48fa-9dbe-481a08c1951d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x"
Jul 17 13:17:08.620679 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:08.620653    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ea86bd-7677-48fa-9dbe-481a08c1951d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x\" (UID: \"72ea86bd-7677-48fa-9dbe-481a08c1951d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x"
Jul 17 13:17:08.708848 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:08.708787    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x"
Jul 17 13:17:08.837915 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:08.837879    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x"]
Jul 17 13:17:08.840227 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:17:08.840203    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ea86bd_7677_48fa_9dbe_481a08c1951d.slice/crio-454b8101ff95cab7a9bb3fb9a2a9522ac98d0a0f11b2435c278375e94458d639 WatchSource:0}: Error finding container 454b8101ff95cab7a9bb3fb9a2a9522ac98d0a0f11b2435c278375e94458d639: Status 404 returned error can't find the container with id 454b8101ff95cab7a9bb3fb9a2a9522ac98d0a0f11b2435c278375e94458d639
Jul 17 13:17:09.211568 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:09.211543    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x" event={"ID":"72ea86bd-7677-48fa-9dbe-481a08c1951d","Type":"ContainerStarted","Data":"7aa2e6cee9a315ef16305615086caa30acf3bb668caa8e2715392a7f00510e71"}
Jul 17 13:17:09.211692 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:09.211588    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x" event={"ID":"72ea86bd-7677-48fa-9dbe-481a08c1951d","Type":"ContainerStarted","Data":"454b8101ff95cab7a9bb3fb9a2a9522ac98d0a0f11b2435c278375e94458d639"}
Jul 17 13:17:09.233971 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:09.233930    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x" podStartSLOduration=1.233915744 podStartE2EDuration="1.233915744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:17:09.232330885 +0000 UTC m=+3304.236377766" watchObservedRunningTime="2026-07-17 13:17:09.233915744 +0000 UTC m=+3304.237962626"
Jul 17 13:17:10.217520 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:10.217483    2569 generic.go:356] "Generic (PLEG): container finished" podID="72ea86bd-7677-48fa-9dbe-481a08c1951d" containerID="7aa2e6cee9a315ef16305615086caa30acf3bb668caa8e2715392a7f00510e71" exitCode=6
Jul 17 13:17:10.217944 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:10.217544    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x" event={"ID":"72ea86bd-7677-48fa-9dbe-481a08c1951d","Type":"ContainerDied","Data":"7aa2e6cee9a315ef16305615086caa30acf3bb668caa8e2715392a7f00510e71"}
Jul 17 13:17:10.217944 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:10.217738    2569 scope.go:117] "RemoveContainer" containerID="7aa2e6cee9a315ef16305615086caa30acf3bb668caa8e2715392a7f00510e71"
Jul 17 13:17:10.918528 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:10.918494    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq"]
Jul 17 13:17:10.921581 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:10.921562    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq"
Jul 17 13:17:10.929949 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:10.929925    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq"]
Jul 17 13:17:11.038325 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:11.038296    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628e121c-175d-4319-adc6-e46236806369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq\" (UID: \"628e121c-175d-4319-adc6-e46236806369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq"
Jul 17 13:17:11.139258 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:11.139198    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628e121c-175d-4319-adc6-e46236806369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq\" (UID: \"628e121c-175d-4319-adc6-e46236806369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq"
Jul 17 13:17:11.139817 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:11.139799    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628e121c-175d-4319-adc6-e46236806369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq\" (UID: \"628e121c-175d-4319-adc6-e46236806369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq"
Jul 17 13:17:11.224547 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:11.224516    2569 generic.go:356] "Generic (PLEG): container finished" podID="72ea86bd-7677-48fa-9dbe-481a08c1951d" containerID="b493dc99ae41eac9c7e19aba66de1974012ff71b96d040b57c8fe1bb9e65f1c8" exitCode=6
Jul 17 13:17:11.224939 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:11.224600    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x" event={"ID":"72ea86bd-7677-48fa-9dbe-481a08c1951d","Type":"ContainerDied","Data":"b493dc99ae41eac9c7e19aba66de1974012ff71b96d040b57c8fe1bb9e65f1c8"}
Jul 17 13:17:11.224939 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:11.224663    2569 scope.go:117] "RemoveContainer" containerID="7aa2e6cee9a315ef16305615086caa30acf3bb668caa8e2715392a7f00510e71"
Jul 17 13:17:11.224939 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:11.224873    2569 scope.go:117] "RemoveContainer" containerID="b493dc99ae41eac9c7e19aba66de1974012ff71b96d040b57c8fe1bb9e65f1c8"
Jul 17 13:17:11.225112 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:17:11.225090    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x_odh-ai-gateway-infra(72ea86bd-7677-48fa-9dbe-481a08c1951d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x" podUID="72ea86bd-7677-48fa-9dbe-481a08c1951d"
Jul 17 13:17:11.232656 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:11.232629    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq"
Jul 17 13:17:11.363323 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:11.363302    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq"]
Jul 17 13:17:11.365620 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:17:11.365596    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628e121c_175d_4319_adc6_e46236806369.slice/crio-5d9081e1523d7ad76e16cf8ecdf40bb43474ccad693bb4dbcb343a2e260edab6 WatchSource:0}: Error finding container 5d9081e1523d7ad76e16cf8ecdf40bb43474ccad693bb4dbcb343a2e260edab6: Status 404 returned error can't find the container with id 5d9081e1523d7ad76e16cf8ecdf40bb43474ccad693bb4dbcb343a2e260edab6
Jul 17 13:17:12.231253 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:12.231227    2569 generic.go:356] "Generic (PLEG): container finished" podID="628e121c-175d-4319-adc6-e46236806369" containerID="069f11253b95c5d58a024957278209bfe1419137f62fc662679b88d06eba9ff2" exitCode=6
Jul 17 13:17:12.231639 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:12.231314    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq" event={"ID":"628e121c-175d-4319-adc6-e46236806369","Type":"ContainerDied","Data":"069f11253b95c5d58a024957278209bfe1419137f62fc662679b88d06eba9ff2"}
Jul 17 13:17:12.231639 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:12.231362    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq" event={"ID":"628e121c-175d-4319-adc6-e46236806369","Type":"ContainerStarted","Data":"5d9081e1523d7ad76e16cf8ecdf40bb43474ccad693bb4dbcb343a2e260edab6"}
Jul 17 13:17:12.231639 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:12.231465    2569 scope.go:117] "RemoveContainer" containerID="069f11253b95c5d58a024957278209bfe1419137f62fc662679b88d06eba9ff2"
Jul 17 13:17:12.233182 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:12.233167    2569 scope.go:117] "RemoveContainer" containerID="b493dc99ae41eac9c7e19aba66de1974012ff71b96d040b57c8fe1bb9e65f1c8"
Jul 17 13:17:12.233337 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:17:12.233304    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x_odh-ai-gateway-infra(72ea86bd-7677-48fa-9dbe-481a08c1951d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x" podUID="72ea86bd-7677-48fa-9dbe-481a08c1951d"
Jul 17 13:17:13.239861 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:13.239821    2569 generic.go:356] "Generic (PLEG): container finished" podID="628e121c-175d-4319-adc6-e46236806369" containerID="9a8ad78dacfcf4d9d5c1e05924d17fe9de5fef03ba2cc101a2a10d5120fcb1b2" exitCode=6
Jul 17 13:17:13.240184 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:13.239910    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq" event={"ID":"628e121c-175d-4319-adc6-e46236806369","Type":"ContainerDied","Data":"9a8ad78dacfcf4d9d5c1e05924d17fe9de5fef03ba2cc101a2a10d5120fcb1b2"}
Jul 17 13:17:13.240184 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:13.240020    2569 scope.go:117] "RemoveContainer" containerID="069f11253b95c5d58a024957278209bfe1419137f62fc662679b88d06eba9ff2"
Jul 17 13:17:13.240279 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:13.240194    2569 scope.go:117] "RemoveContainer" containerID="9a8ad78dacfcf4d9d5c1e05924d17fe9de5fef03ba2cc101a2a10d5120fcb1b2"
Jul 17 13:17:13.240379 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:17:13.240360    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq_odh-ai-gateway-infra(628e121c-175d-4319-adc6-e46236806369)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq" podUID="628e121c-175d-4319-adc6-e46236806369"
Jul 17 13:17:13.462500 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:13.462411    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 13:17:13.481041 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:13.468171    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 13:17:14.249250 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:14.249224    2569 scope.go:117] "RemoveContainer" containerID="9a8ad78dacfcf4d9d5c1e05924d17fe9de5fef03ba2cc101a2a10d5120fcb1b2"
Jul 17 13:17:14.249617 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:17:14.249394    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq_odh-ai-gateway-infra(628e121c-175d-4319-adc6-e46236806369)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq" podUID="628e121c-175d-4319-adc6-e46236806369"
Jul 17 13:17:14.628631 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:14.628566    2569 scope.go:117] "RemoveContainer" containerID="a47e8bd5ab0b6d1cbb77581295bcc8a3d80f3ffa460c02aff5e7d529e759df7a"
Jul 17 13:17:15.255532 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:15.255508    2569 generic.go:356] "Generic (PLEG): container finished" podID="ed4d00a1-2462-4afb-b629-70bdecdda560" containerID="8dd8fc3c878875e136d2b27f5e5f69fecc1e95d3d3fb4e0b008a9d29186216ff" exitCode=6
Jul 17 13:17:15.255846 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:15.255583    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l" event={"ID":"ed4d00a1-2462-4afb-b629-70bdecdda560","Type":"ContainerDied","Data":"8dd8fc3c878875e136d2b27f5e5f69fecc1e95d3d3fb4e0b008a9d29186216ff"}
Jul 17 13:17:15.255846 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:15.255660    2569 scope.go:117] "RemoveContainer" containerID="a47e8bd5ab0b6d1cbb77581295bcc8a3d80f3ffa460c02aff5e7d529e759df7a"
Jul 17 13:17:15.255941 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:15.255916    2569 scope.go:117] "RemoveContainer" containerID="8dd8fc3c878875e136d2b27f5e5f69fecc1e95d3d3fb4e0b008a9d29186216ff"
Jul 17 13:17:15.256103 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:17:15.256086    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l_odh-ai-gateway-infra(ed4d00a1-2462-4afb-b629-70bdecdda560)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l" podUID="ed4d00a1-2462-4afb-b629-70bdecdda560"
Jul 17 13:17:16.290203 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:16.290165    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l"]
Jul 17 13:17:16.429370 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:16.429348    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l"
Jul 17 13:17:16.581029 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:16.580959    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4d00a1-2462-4afb-b629-70bdecdda560-maas-api-service-ca\") pod \"ed4d00a1-2462-4afb-b629-70bdecdda560\" (UID: \"ed4d00a1-2462-4afb-b629-70bdecdda560\") "
Jul 17 13:17:16.581271 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:16.581251    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed4d00a1-2462-4afb-b629-70bdecdda560-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ed4d00a1-2462-4afb-b629-70bdecdda560" (UID: "ed4d00a1-2462-4afb-b629-70bdecdda560"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:17:16.682417 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:16.682391    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed4d00a1-2462-4afb-b629-70bdecdda560-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:17:17.269424 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:17.269391    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l"
Jul 17 13:17:17.269644 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:17.269437    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l" event={"ID":"ed4d00a1-2462-4afb-b629-70bdecdda560","Type":"ContainerDied","Data":"46851b75f5354bc9ff95c19a105ddd90f51bc1e3dca0ebd975598de93deaec99"}
Jul 17 13:17:17.269644 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:17.269520    2569 scope.go:117] "RemoveContainer" containerID="8dd8fc3c878875e136d2b27f5e5f69fecc1e95d3d3fb4e0b008a9d29186216ff"
Jul 17 13:17:17.291385 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:17.291350    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l"]
Jul 17 13:17:17.293763 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:17.293724    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n5z7l"]
Jul 17 13:17:17.634119 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:17.634060    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed4d00a1-2462-4afb-b629-70bdecdda560" path="/var/lib/kubelet/pods/ed4d00a1-2462-4afb-b629-70bdecdda560/volumes"
Jul 17 13:17:20.628256 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:20.628224    2569 scope.go:117] "RemoveContainer" containerID="335ba0976bee2fa0919a5287be21beceb0800fe3bff93c47f003299c8c756cda"
Jul 17 13:17:21.292319 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:21.292286    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7" event={"ID":"fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3","Type":"ContainerStarted","Data":"6217cf74bf614d4ee4c17a516f8d21993e5b167d9f2b1e58dca73c13ed8f2918"}
Jul 17 13:17:21.310141 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:21.310094    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7" podStartSLOduration=19.310079339 podStartE2EDuration="19.310079339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:17:21.307731154 +0000 UTC m=+3316.311778053" watchObservedRunningTime="2026-07-17 13:17:21.310079339 +0000 UTC m=+3316.314126220"
Jul 17 13:17:22.298352 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:22.298323    2569 generic.go:356] "Generic (PLEG): container finished" podID="fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3" containerID="6217cf74bf614d4ee4c17a516f8d21993e5b167d9f2b1e58dca73c13ed8f2918" exitCode=6
Jul 17 13:17:22.298800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:22.298391    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7" event={"ID":"fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3","Type":"ContainerDied","Data":"6217cf74bf614d4ee4c17a516f8d21993e5b167d9f2b1e58dca73c13ed8f2918"}
Jul 17 13:17:22.298800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:22.298450    2569 scope.go:117] "RemoveContainer" containerID="335ba0976bee2fa0919a5287be21beceb0800fe3bff93c47f003299c8c756cda"
Jul 17 13:17:22.298800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:22.298688    2569 scope.go:117] "RemoveContainer" containerID="6217cf74bf614d4ee4c17a516f8d21993e5b167d9f2b1e58dca73c13ed8f2918"
Jul 17 13:17:22.298947 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:17:22.298913    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7_odh-ai-gateway-infra(fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7" podUID="fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3"
Jul 17 13:17:22.326239 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:22.326213    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7"]
Jul 17 13:17:23.442393 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:23.442364    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7"
Jul 17 13:17:23.527613 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:23.527583    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3-maas-api-service-ca\") pod \"fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3\" (UID: \"fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3\") "
Jul 17 13:17:23.527922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:23.527899    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3" (UID: "fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:17:23.628787 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:23.628704    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:17:23.629287 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:23.629269    2569 scope.go:117] "RemoveContainer" containerID="b493dc99ae41eac9c7e19aba66de1974012ff71b96d040b57c8fe1bb9e65f1c8"
Jul 17 13:17:24.318656 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:24.318627    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7"
Jul 17 13:17:24.318656 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:24.318641    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7" event={"ID":"fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3","Type":"ContainerDied","Data":"e88484bdcac8080a827350b0d2fa452ca327e48464b51dd1e46a45ed834370e9"}
Jul 17 13:17:24.318873 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:24.318737    2569 scope.go:117] "RemoveContainer" containerID="6217cf74bf614d4ee4c17a516f8d21993e5b167d9f2b1e58dca73c13ed8f2918"
Jul 17 13:17:24.321399 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:24.321372    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x" event={"ID":"72ea86bd-7677-48fa-9dbe-481a08c1951d","Type":"ContainerStarted","Data":"6aa4400d98123d5d17b8a0ad762b00a1c08d09803ad06ab07565c5cbfa0446a5"}
Jul 17 13:17:24.359033 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:24.359011    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7"]
Jul 17 13:17:24.366939 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:24.366920    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-6s2q7"]
Jul 17 13:17:25.329459 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:25.329431    2569 generic.go:356] "Generic (PLEG): container finished" podID="72ea86bd-7677-48fa-9dbe-481a08c1951d" containerID="6aa4400d98123d5d17b8a0ad762b00a1c08d09803ad06ab07565c5cbfa0446a5" exitCode=6
Jul 17 13:17:25.329875 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:25.329482    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x" event={"ID":"72ea86bd-7677-48fa-9dbe-481a08c1951d","Type":"ContainerDied","Data":"6aa4400d98123d5d17b8a0ad762b00a1c08d09803ad06ab07565c5cbfa0446a5"}
Jul 17 13:17:25.329875 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:25.329533    2569 scope.go:117] "RemoveContainer" containerID="b493dc99ae41eac9c7e19aba66de1974012ff71b96d040b57c8fe1bb9e65f1c8"
Jul 17 13:17:25.329875 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:25.329827    2569 scope.go:117] "RemoveContainer" containerID="6aa4400d98123d5d17b8a0ad762b00a1c08d09803ad06ab07565c5cbfa0446a5"
Jul 17 13:17:25.330087 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:17:25.330060    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x_odh-ai-gateway-infra(72ea86bd-7677-48fa-9dbe-481a08c1951d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x" podUID="72ea86bd-7677-48fa-9dbe-481a08c1951d"
Jul 17 13:17:25.357792 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:25.357769    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x"]
Jul 17 13:17:25.633244 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:25.633172    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3" path="/var/lib/kubelet/pods/fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3/volumes"
Jul 17 13:17:26.473785 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:26.473745    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x"
Jul 17 13:17:26.548739 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:26.548712    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ea86bd-7677-48fa-9dbe-481a08c1951d-maas-api-service-ca\") pod \"72ea86bd-7677-48fa-9dbe-481a08c1951d\" (UID: \"72ea86bd-7677-48fa-9dbe-481a08c1951d\") "
Jul 17 13:17:26.549086 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:26.549065    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ea86bd-7677-48fa-9dbe-481a08c1951d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "72ea86bd-7677-48fa-9dbe-481a08c1951d" (UID: "72ea86bd-7677-48fa-9dbe-481a08c1951d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:17:26.628297 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:26.628248    2569 scope.go:117] "RemoveContainer" containerID="9a8ad78dacfcf4d9d5c1e05924d17fe9de5fef03ba2cc101a2a10d5120fcb1b2"
Jul 17 13:17:26.649735 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:26.649715    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ea86bd-7677-48fa-9dbe-481a08c1951d-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:17:27.342580 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:27.342553    2569 generic.go:356] "Generic (PLEG): container finished" podID="628e121c-175d-4319-adc6-e46236806369" containerID="c05e7baeb75b182073ea67c887abd0d3982f6449d18a6db35ceb0de26cd06518" exitCode=6
Jul 17 13:17:27.342695 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:27.342616    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq" event={"ID":"628e121c-175d-4319-adc6-e46236806369","Type":"ContainerDied","Data":"c05e7baeb75b182073ea67c887abd0d3982f6449d18a6db35ceb0de26cd06518"}
Jul 17 13:17:27.342695 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:27.342684    2569 scope.go:117] "RemoveContainer" containerID="9a8ad78dacfcf4d9d5c1e05924d17fe9de5fef03ba2cc101a2a10d5120fcb1b2"
Jul 17 13:17:27.343028 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:27.343006    2569 scope.go:117] "RemoveContainer" containerID="c05e7baeb75b182073ea67c887abd0d3982f6449d18a6db35ceb0de26cd06518"
Jul 17 13:17:27.343225 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:17:27.343205    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq_odh-ai-gateway-infra(628e121c-175d-4319-adc6-e46236806369)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq" podUID="628e121c-175d-4319-adc6-e46236806369"
Jul 17 13:17:27.344109 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:27.344044    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x"
Jul 17 13:17:27.344109 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:27.344070    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x" event={"ID":"72ea86bd-7677-48fa-9dbe-481a08c1951d","Type":"ContainerDied","Data":"454b8101ff95cab7a9bb3fb9a2a9522ac98d0a0f11b2435c278375e94458d639"}
Jul 17 13:17:27.353835 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:27.353818    2569 scope.go:117] "RemoveContainer" containerID="6aa4400d98123d5d17b8a0ad762b00a1c08d09803ad06ab07565c5cbfa0446a5"
Jul 17 13:17:27.373595 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:27.373573    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x"]
Jul 17 13:17:27.376987 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:27.376965    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9h79x"]
Jul 17 13:17:27.634109 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:27.634089    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72ea86bd-7677-48fa-9dbe-481a08c1951d" path="/var/lib/kubelet/pods/72ea86bd-7677-48fa-9dbe-481a08c1951d/volumes"
Jul 17 13:17:28.369635 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:28.369606    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq"]
Jul 17 13:17:28.505642 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:28.505617    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq"
Jul 17 13:17:28.562051 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:28.562029    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628e121c-175d-4319-adc6-e46236806369-maas-api-service-ca\") pod \"628e121c-175d-4319-adc6-e46236806369\" (UID: \"628e121c-175d-4319-adc6-e46236806369\") "
Jul 17 13:17:28.562350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:28.562326    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/628e121c-175d-4319-adc6-e46236806369-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "628e121c-175d-4319-adc6-e46236806369" (UID: "628e121c-175d-4319-adc6-e46236806369"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:17:28.663496 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:28.663430    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628e121c-175d-4319-adc6-e46236806369-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:17:29.357774 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:29.357722    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq"
Jul 17 13:17:29.357975 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:29.357777    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq" event={"ID":"628e121c-175d-4319-adc6-e46236806369","Type":"ContainerDied","Data":"5d9081e1523d7ad76e16cf8ecdf40bb43474ccad693bb4dbcb343a2e260edab6"}
Jul 17 13:17:29.357975 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:29.357854    2569 scope.go:117] "RemoveContainer" containerID="c05e7baeb75b182073ea67c887abd0d3982f6449d18a6db35ceb0de26cd06518"
Jul 17 13:17:29.382441 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:29.382415    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq"]
Jul 17 13:17:29.388956 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:29.388933    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-czrlq"]
Jul 17 13:17:29.633793 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:29.633722    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="628e121c-175d-4319-adc6-e46236806369" path="/var/lib/kubelet/pods/628e121c-175d-4319-adc6-e46236806369/volumes"
Jul 17 13:17:49.772398 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772361    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q"]
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772681    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed4d00a1-2462-4afb-b629-70bdecdda560" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772693    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed4d00a1-2462-4afb-b629-70bdecdda560" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772703    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772709    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772717    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="628e121c-175d-4319-adc6-e46236806369" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772724    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="628e121c-175d-4319-adc6-e46236806369" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772734    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772740    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772745    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="628e121c-175d-4319-adc6-e46236806369" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772768    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="628e121c-175d-4319-adc6-e46236806369" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772782    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ea86bd-7677-48fa-9dbe-481a08c1951d" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772788    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ea86bd-7677-48fa-9dbe-481a08c1951d" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772794    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ea86bd-7677-48fa-9dbe-481a08c1951d" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772800    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ea86bd-7677-48fa-9dbe-481a08c1951d" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772807    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed4d00a1-2462-4afb-b629-70bdecdda560" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772812    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed4d00a1-2462-4afb-b629-70bdecdda560" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772822    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ea86bd-7677-48fa-9dbe-481a08c1951d" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772827    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ea86bd-7677-48fa-9dbe-481a08c1951d" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772888    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ea86bd-7677-48fa-9dbe-481a08c1951d" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772896    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="628e121c-175d-4319-adc6-e46236806369" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772903    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed4d00a1-2462-4afb-b629-70bdecdda560" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772910    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed4d00a1-2462-4afb-b629-70bdecdda560" containerName="revoke-keys"
Jul 17 13:17:49.772902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772916    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ea86bd-7677-48fa-9dbe-481a08c1951d" containerName="revoke-keys"
Jul 17 13:17:49.773625 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772922    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="628e121c-175d-4319-adc6-e46236806369" containerName="revoke-keys"
Jul 17 13:17:49.773625 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772929    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3" containerName="revoke-keys"
Jul 17 13:17:49.773625 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.772934    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3" containerName="revoke-keys"
Jul 17 13:17:49.775670 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.775650    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q"
Jul 17 13:17:49.778539 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.778512    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:17:49.778709 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.778576    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:17:49.783377 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.782974    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q"]
Jul 17 13:17:49.817538 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.817506    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938ae4de-6532-4720-be35-6d1c8b21093f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q\" (UID: \"938ae4de-6532-4720-be35-6d1c8b21093f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q"
Jul 17 13:17:49.918643 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.918620    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938ae4de-6532-4720-be35-6d1c8b21093f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q\" (UID: \"938ae4de-6532-4720-be35-6d1c8b21093f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q"
Jul 17 13:17:49.919153 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:49.919135    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938ae4de-6532-4720-be35-6d1c8b21093f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q\" (UID: \"938ae4de-6532-4720-be35-6d1c8b21093f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q"
Jul 17 13:17:50.088149 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:50.088080    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q"
Jul 17 13:17:50.213348 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:50.213323    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q"]
Jul 17 13:17:50.215647 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:17:50.215620    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938ae4de_6532_4720_be35_6d1c8b21093f.slice/crio-bd5f3697cb85d150a2c762e006359d97a006e87dde630fd7b91ceaf923f88c06 WatchSource:0}: Error finding container bd5f3697cb85d150a2c762e006359d97a006e87dde630fd7b91ceaf923f88c06: Status 404 returned error can't find the container with id bd5f3697cb85d150a2c762e006359d97a006e87dde630fd7b91ceaf923f88c06
Jul 17 13:17:50.462049 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:50.462024    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q" event={"ID":"938ae4de-6532-4720-be35-6d1c8b21093f","Type":"ContainerStarted","Data":"70ab36de91dbee998bcb08b1cd47a94f0ace65756fcac3d00170ab360ad9e870"}
Jul 17 13:17:50.462145 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:50.462109    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q" event={"ID":"938ae4de-6532-4720-be35-6d1c8b21093f","Type":"ContainerStarted","Data":"bd5f3697cb85d150a2c762e006359d97a006e87dde630fd7b91ceaf923f88c06"}
Jul 17 13:17:50.480261 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:50.480213    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q" podStartSLOduration=1.4801962020000001 podStartE2EDuration="1.480196202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:17:50.479883994 +0000 UTC m=+3345.483930881" watchObservedRunningTime="2026-07-17 13:17:50.480196202 +0000 UTC m=+3345.484243085"
Jul 17 13:17:51.469606 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:51.469570    2569 generic.go:356] "Generic (PLEG): container finished" podID="938ae4de-6532-4720-be35-6d1c8b21093f" containerID="70ab36de91dbee998bcb08b1cd47a94f0ace65756fcac3d00170ab360ad9e870" exitCode=6
Jul 17 13:17:51.469994 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:51.469659    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q" event={"ID":"938ae4de-6532-4720-be35-6d1c8b21093f","Type":"ContainerDied","Data":"70ab36de91dbee998bcb08b1cd47a94f0ace65756fcac3d00170ab360ad9e870"}
Jul 17 13:17:51.469994 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:51.469843    2569 scope.go:117] "RemoveContainer" containerID="70ab36de91dbee998bcb08b1cd47a94f0ace65756fcac3d00170ab360ad9e870"
Jul 17 13:17:52.476159 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:52.476127    2569 generic.go:356] "Generic (PLEG): container finished" podID="938ae4de-6532-4720-be35-6d1c8b21093f" containerID="286c0a08d14cf4c6ec21f532f25b6ff07318ec8c82dc00bf801ec6735f09e7b7" exitCode=6
Jul 17 13:17:52.476578 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:52.476199    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q" event={"ID":"938ae4de-6532-4720-be35-6d1c8b21093f","Type":"ContainerDied","Data":"286c0a08d14cf4c6ec21f532f25b6ff07318ec8c82dc00bf801ec6735f09e7b7"}
Jul 17 13:17:52.476578 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:52.476262    2569 scope.go:117] "RemoveContainer" containerID="70ab36de91dbee998bcb08b1cd47a94f0ace65756fcac3d00170ab360ad9e870"
Jul 17 13:17:52.476578 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:52.476502    2569 scope.go:117] "RemoveContainer" containerID="286c0a08d14cf4c6ec21f532f25b6ff07318ec8c82dc00bf801ec6735f09e7b7"
Jul 17 13:17:52.476794 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:17:52.476742    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q_odh-ai-gateway-infra(938ae4de-6532-4720-be35-6d1c8b21093f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q" podUID="938ae4de-6532-4720-be35-6d1c8b21093f"
Jul 17 13:17:53.482929 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:17:53.482903    2569 scope.go:117] "RemoveContainer" containerID="286c0a08d14cf4c6ec21f532f25b6ff07318ec8c82dc00bf801ec6735f09e7b7"
Jul 17 13:17:53.483304 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:17:53.483064    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q_odh-ai-gateway-infra(938ae4de-6532-4720-be35-6d1c8b21093f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q" podUID="938ae4de-6532-4720-be35-6d1c8b21093f"
Jul 17 13:18:00.962149 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:00.962108    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl"]
Jul 17 13:18:00.962613 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:00.962591    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3" containerName="revoke-keys"
Jul 17 13:18:00.962613 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:00.962609    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3" containerName="revoke-keys"
Jul 17 13:18:00.962743 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:00.962620    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="628e121c-175d-4319-adc6-e46236806369" containerName="revoke-keys"
Jul 17 13:18:00.962743 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:00.962629    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="628e121c-175d-4319-adc6-e46236806369" containerName="revoke-keys"
Jul 17 13:18:00.962743 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:00.962651    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed4d00a1-2462-4afb-b629-70bdecdda560" containerName="revoke-keys"
Jul 17 13:18:00.962743 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:00.962660    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed4d00a1-2462-4afb-b629-70bdecdda560" containerName="revoke-keys"
Jul 17 13:18:00.962972 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:00.962788    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe07ce37-8dc6-498d-b0fa-9c67d3c3e7e3" containerName="revoke-keys"
Jul 17 13:18:00.962972 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:00.962805    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed4d00a1-2462-4afb-b629-70bdecdda560" containerName="revoke-keys"
Jul 17 13:18:00.962972 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:00.962814    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ea86bd-7677-48fa-9dbe-481a08c1951d" containerName="revoke-keys"
Jul 17 13:18:00.962972 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:00.962826    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="628e121c-175d-4319-adc6-e46236806369" containerName="revoke-keys"
Jul 17 13:18:00.966958 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:00.966936    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl"
Jul 17 13:18:00.973096 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:00.973063    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl"]
Jul 17 13:18:01.002516 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:01.002493    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67273611-969c-419b-8a2c-f1da39bb4679-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl\" (UID: \"67273611-969c-419b-8a2c-f1da39bb4679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl"
Jul 17 13:18:01.103758 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:01.103733    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67273611-969c-419b-8a2c-f1da39bb4679-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl\" (UID: \"67273611-969c-419b-8a2c-f1da39bb4679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl"
Jul 17 13:18:01.104354 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:01.104335    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67273611-969c-419b-8a2c-f1da39bb4679-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl\" (UID: \"67273611-969c-419b-8a2c-f1da39bb4679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl"
Jul 17 13:18:01.278018 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:01.277996    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl"
Jul 17 13:18:01.611299 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:01.611225    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl"]
Jul 17 13:18:01.614554 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:18:01.614529    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67273611_969c_419b_8a2c_f1da39bb4679.slice/crio-8453410e76395a80354e913cc0b8c21832fcf2968f0ead249b4151b943a2838b WatchSource:0}: Error finding container 8453410e76395a80354e913cc0b8c21832fcf2968f0ead249b4151b943a2838b: Status 404 returned error can't find the container with id 8453410e76395a80354e913cc0b8c21832fcf2968f0ead249b4151b943a2838b
Jul 17 13:18:02.115521 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:02.115496    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq"]
Jul 17 13:18:02.118741 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:02.118725    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq"
Jul 17 13:18:02.126178 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:02.126151    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq"]
Jul 17 13:18:02.213945 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:02.213877    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c0d9f03-e1b9-42e7-a808-afa4de60bce5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq\" (UID: \"4c0d9f03-e1b9-42e7-a808-afa4de60bce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq"
Jul 17 13:18:02.315183 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:02.315156    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c0d9f03-e1b9-42e7-a808-afa4de60bce5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq\" (UID: \"4c0d9f03-e1b9-42e7-a808-afa4de60bce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq"
Jul 17 13:18:02.315716 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:02.315697    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c0d9f03-e1b9-42e7-a808-afa4de60bce5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq\" (UID: \"4c0d9f03-e1b9-42e7-a808-afa4de60bce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq"
Jul 17 13:18:02.429626 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:02.429601    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq"
Jul 17 13:18:02.526475 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:02.526442    2569 generic.go:356] "Generic (PLEG): container finished" podID="67273611-969c-419b-8a2c-f1da39bb4679" containerID="39331bbcace67cd379986447a18f2af4ea7bf021a405bb07156c72032598992d" exitCode=6
Jul 17 13:18:02.526620 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:02.526536    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl" event={"ID":"67273611-969c-419b-8a2c-f1da39bb4679","Type":"ContainerDied","Data":"39331bbcace67cd379986447a18f2af4ea7bf021a405bb07156c72032598992d"}
Jul 17 13:18:02.526682 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:02.526636    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl" event={"ID":"67273611-969c-419b-8a2c-f1da39bb4679","Type":"ContainerStarted","Data":"8453410e76395a80354e913cc0b8c21832fcf2968f0ead249b4151b943a2838b"}
Jul 17 13:18:02.526767 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:02.526736    2569 scope.go:117] "RemoveContainer" containerID="39331bbcace67cd379986447a18f2af4ea7bf021a405bb07156c72032598992d"
Jul 17 13:18:02.563165 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:02.563044    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq"]
Jul 17 13:18:02.565151 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:18:02.565127    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0d9f03_e1b9_42e7_a808_afa4de60bce5.slice/crio-5ab69851f7f71ce676132209f2f6e91faab58c23a93afcd0a497446659d1a4fb WatchSource:0}: Error finding container 5ab69851f7f71ce676132209f2f6e91faab58c23a93afcd0a497446659d1a4fb: Status 404 returned error can't find the container with id 5ab69851f7f71ce676132209f2f6e91faab58c23a93afcd0a497446659d1a4fb
Jul 17 13:18:03.535769 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:03.535722    2569 generic.go:356] "Generic (PLEG): container finished" podID="67273611-969c-419b-8a2c-f1da39bb4679" containerID="396538e76eb4ff0f258c9829be7c43d87b2a08787aa5f2bea8459b6472906376" exitCode=6
Jul 17 13:18:03.536164 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:03.535819    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl" event={"ID":"67273611-969c-419b-8a2c-f1da39bb4679","Type":"ContainerDied","Data":"396538e76eb4ff0f258c9829be7c43d87b2a08787aa5f2bea8459b6472906376"}
Jul 17 13:18:03.536164 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:03.535880    2569 scope.go:117] "RemoveContainer" containerID="39331bbcace67cd379986447a18f2af4ea7bf021a405bb07156c72032598992d"
Jul 17 13:18:03.536164 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:03.536093    2569 scope.go:117] "RemoveContainer" containerID="396538e76eb4ff0f258c9829be7c43d87b2a08787aa5f2bea8459b6472906376"
Jul 17 13:18:03.536317 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:18:03.536264    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl_odh-ai-gateway-infra(67273611-969c-419b-8a2c-f1da39bb4679)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl" podUID="67273611-969c-419b-8a2c-f1da39bb4679"
Jul 17 13:18:03.537366 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:03.537347    2569 generic.go:356] "Generic (PLEG): container finished" podID="4c0d9f03-e1b9-42e7-a808-afa4de60bce5" containerID="c5609616ebbf6b98127f93d8ddbc431b3f9fbeaa912deb0ebea7004f3419190b" exitCode=6
Jul 17 13:18:03.537447 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:03.537378    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq" event={"ID":"4c0d9f03-e1b9-42e7-a808-afa4de60bce5","Type":"ContainerDied","Data":"c5609616ebbf6b98127f93d8ddbc431b3f9fbeaa912deb0ebea7004f3419190b"}
Jul 17 13:18:03.537447 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:03.537431    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq" event={"ID":"4c0d9f03-e1b9-42e7-a808-afa4de60bce5","Type":"ContainerStarted","Data":"5ab69851f7f71ce676132209f2f6e91faab58c23a93afcd0a497446659d1a4fb"}
Jul 17 13:18:03.537606 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:03.537591    2569 scope.go:117] "RemoveContainer" containerID="c5609616ebbf6b98127f93d8ddbc431b3f9fbeaa912deb0ebea7004f3419190b"
Jul 17 13:18:04.544776 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:04.544729    2569 scope.go:117] "RemoveContainer" containerID="396538e76eb4ff0f258c9829be7c43d87b2a08787aa5f2bea8459b6472906376"
Jul 17 13:18:04.545193 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:18:04.544940    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl_odh-ai-gateway-infra(67273611-969c-419b-8a2c-f1da39bb4679)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl" podUID="67273611-969c-419b-8a2c-f1da39bb4679"
Jul 17 13:18:04.547204 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:04.547178    2569 generic.go:356] "Generic (PLEG): container finished" podID="4c0d9f03-e1b9-42e7-a808-afa4de60bce5" containerID="9224204e308ef8eabd1d796bc3e1ea815d2aea64a9693becd9176f8548eccc12" exitCode=6
Jul 17 13:18:04.547331 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:04.547249    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq" event={"ID":"4c0d9f03-e1b9-42e7-a808-afa4de60bce5","Type":"ContainerDied","Data":"9224204e308ef8eabd1d796bc3e1ea815d2aea64a9693becd9176f8548eccc12"}
Jul 17 13:18:04.547331 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:04.547303    2569 scope.go:117] "RemoveContainer" containerID="c5609616ebbf6b98127f93d8ddbc431b3f9fbeaa912deb0ebea7004f3419190b"
Jul 17 13:18:04.547478 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:04.547455    2569 scope.go:117] "RemoveContainer" containerID="9224204e308ef8eabd1d796bc3e1ea815d2aea64a9693becd9176f8548eccc12"
Jul 17 13:18:04.547664 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:18:04.547642    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq_odh-ai-gateway-infra(4c0d9f03-e1b9-42e7-a808-afa4de60bce5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq" podUID="4c0d9f03-e1b9-42e7-a808-afa4de60bce5"
Jul 17 13:18:04.628913 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:04.628896    2569 scope.go:117] "RemoveContainer" containerID="286c0a08d14cf4c6ec21f532f25b6ff07318ec8c82dc00bf801ec6735f09e7b7"
Jul 17 13:18:05.553437 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:05.553415    2569 generic.go:356] "Generic (PLEG): container finished" podID="938ae4de-6532-4720-be35-6d1c8b21093f" containerID="5f59cac56b978e48bdccac6653e2d0093f3b0e2cf138ebef8e87c5bee90a6655" exitCode=6
Jul 17 13:18:05.553803 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:05.553502    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q" event={"ID":"938ae4de-6532-4720-be35-6d1c8b21093f","Type":"ContainerDied","Data":"5f59cac56b978e48bdccac6653e2d0093f3b0e2cf138ebef8e87c5bee90a6655"}
Jul 17 13:18:05.553803 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:05.553573    2569 scope.go:117] "RemoveContainer" containerID="286c0a08d14cf4c6ec21f532f25b6ff07318ec8c82dc00bf801ec6735f09e7b7"
Jul 17 13:18:05.553929 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:05.553838    2569 scope.go:117] "RemoveContainer" containerID="5f59cac56b978e48bdccac6653e2d0093f3b0e2cf138ebef8e87c5bee90a6655"
Jul 17 13:18:05.554058 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:18:05.554033    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q_odh-ai-gateway-infra(938ae4de-6532-4720-be35-6d1c8b21093f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q" podUID="938ae4de-6532-4720-be35-6d1c8b21093f"
Jul 17 13:18:05.555352 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:05.555336    2569 scope.go:117] "RemoveContainer" containerID="9224204e308ef8eabd1d796bc3e1ea815d2aea64a9693becd9176f8548eccc12"
Jul 17 13:18:05.555478 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:18:05.555462    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq_odh-ai-gateway-infra(4c0d9f03-e1b9-42e7-a808-afa4de60bce5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq" podUID="4c0d9f03-e1b9-42e7-a808-afa4de60bce5"
Jul 17 13:18:06.580594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:06.580562    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q"]
Jul 17 13:18:06.716486 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:06.716466    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q"
Jul 17 13:18:06.844295 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:06.844230    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938ae4de-6532-4720-be35-6d1c8b21093f-maas-api-service-ca\") pod \"938ae4de-6532-4720-be35-6d1c8b21093f\" (UID: \"938ae4de-6532-4720-be35-6d1c8b21093f\") "
Jul 17 13:18:06.844580 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:06.844559    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938ae4de-6532-4720-be35-6d1c8b21093f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "938ae4de-6532-4720-be35-6d1c8b21093f" (UID: "938ae4de-6532-4720-be35-6d1c8b21093f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:18:06.945177 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:06.945153    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/938ae4de-6532-4720-be35-6d1c8b21093f-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:18:07.567173 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:07.567146    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q"
Jul 17 13:18:07.567324 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:07.567181    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q" event={"ID":"938ae4de-6532-4720-be35-6d1c8b21093f","Type":"ContainerDied","Data":"bd5f3697cb85d150a2c762e006359d97a006e87dde630fd7b91ceaf923f88c06"}
Jul 17 13:18:07.567324 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:07.567245    2569 scope.go:117] "RemoveContainer" containerID="5f59cac56b978e48bdccac6653e2d0093f3b0e2cf138ebef8e87c5bee90a6655"
Jul 17 13:18:07.588949 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:07.588928    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q"]
Jul 17 13:18:07.594493 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:07.594472    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-vcw7q"]
Jul 17 13:18:07.633652 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:07.633631    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="938ae4de-6532-4720-be35-6d1c8b21093f" path="/var/lib/kubelet/pods/938ae4de-6532-4720-be35-6d1c8b21093f/volumes"
Jul 17 13:18:14.267425 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:14.267394    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:18:16.628930 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:16.628897    2569 scope.go:117] "RemoveContainer" containerID="396538e76eb4ff0f258c9829be7c43d87b2a08787aa5f2bea8459b6472906376"
Jul 17 13:18:17.633547 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:17.633472    2569 generic.go:356] "Generic (PLEG): container finished" podID="67273611-969c-419b-8a2c-f1da39bb4679" containerID="23d4c37a54781afc1e912b1f8c5f5ff07e3da83f269cfedecf96b92c47b73a96" exitCode=6
Jul 17 13:18:17.636036 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:17.636013    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl" event={"ID":"67273611-969c-419b-8a2c-f1da39bb4679","Type":"ContainerDied","Data":"23d4c37a54781afc1e912b1f8c5f5ff07e3da83f269cfedecf96b92c47b73a96"}
Jul 17 13:18:17.636150 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:17.636069    2569 scope.go:117] "RemoveContainer" containerID="396538e76eb4ff0f258c9829be7c43d87b2a08787aa5f2bea8459b6472906376"
Jul 17 13:18:17.636328 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:17.636310    2569 scope.go:117] "RemoveContainer" containerID="23d4c37a54781afc1e912b1f8c5f5ff07e3da83f269cfedecf96b92c47b73a96"
Jul 17 13:18:17.636518 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:18:17.636497    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl_odh-ai-gateway-infra(67273611-969c-419b-8a2c-f1da39bb4679)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl" podUID="67273611-969c-419b-8a2c-f1da39bb4679"
Jul 17 13:18:18.662608 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:18.662567    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl"]
Jul 17 13:18:18.802671 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:18.802645    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl"
Jul 17 13:18:18.938724 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:18.938660    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67273611-969c-419b-8a2c-f1da39bb4679-maas-api-service-ca\") pod \"67273611-969c-419b-8a2c-f1da39bb4679\" (UID: \"67273611-969c-419b-8a2c-f1da39bb4679\") "
Jul 17 13:18:18.939031 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:18.939009    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67273611-969c-419b-8a2c-f1da39bb4679-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "67273611-969c-419b-8a2c-f1da39bb4679" (UID: "67273611-969c-419b-8a2c-f1da39bb4679"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:18:19.039598 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:19.039572    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67273611-969c-419b-8a2c-f1da39bb4679-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:18:19.628639 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:19.628610    2569 scope.go:117] "RemoveContainer" containerID="9224204e308ef8eabd1d796bc3e1ea815d2aea64a9693becd9176f8548eccc12"
Jul 17 13:18:19.646288 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:19.646246    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl" event={"ID":"67273611-969c-419b-8a2c-f1da39bb4679","Type":"ContainerDied","Data":"8453410e76395a80354e913cc0b8c21832fcf2968f0ead249b4151b943a2838b"}
Jul 17 13:18:19.646404 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:19.646327    2569 scope.go:117] "RemoveContainer" containerID="23d4c37a54781afc1e912b1f8c5f5ff07e3da83f269cfedecf96b92c47b73a96"
Jul 17 13:18:19.646404 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:19.646265    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl"
Jul 17 13:18:19.681775 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:19.681738    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl"]
Jul 17 13:18:19.685722 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:19.685700    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qbccl"]
Jul 17 13:18:20.653939 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:20.653917    2569 generic.go:356] "Generic (PLEG): container finished" podID="4c0d9f03-e1b9-42e7-a808-afa4de60bce5" containerID="2fc93b294987fc894213cb030373a7652b46ee0e663c8e15c32f8759a5098394" exitCode=6
Jul 17 13:18:20.654081 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:20.653946    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq" event={"ID":"4c0d9f03-e1b9-42e7-a808-afa4de60bce5","Type":"ContainerDied","Data":"2fc93b294987fc894213cb030373a7652b46ee0e663c8e15c32f8759a5098394"}
Jul 17 13:18:20.654081 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:20.653995    2569 scope.go:117] "RemoveContainer" containerID="9224204e308ef8eabd1d796bc3e1ea815d2aea64a9693becd9176f8548eccc12"
Jul 17 13:18:20.654226 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:20.654209    2569 scope.go:117] "RemoveContainer" containerID="2fc93b294987fc894213cb030373a7652b46ee0e663c8e15c32f8759a5098394"
Jul 17 13:18:20.654402 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:18:20.654385    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq_odh-ai-gateway-infra(4c0d9f03-e1b9-42e7-a808-afa4de60bce5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq" podUID="4c0d9f03-e1b9-42e7-a808-afa4de60bce5"
Jul 17 13:18:21.635314 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:21.635280    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67273611-969c-419b-8a2c-f1da39bb4679" path="/var/lib/kubelet/pods/67273611-969c-419b-8a2c-f1da39bb4679/volumes"
Jul 17 13:18:21.680886 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:21.680853    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq"]
Jul 17 13:18:21.808558 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:21.808535    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq"
Jul 17 13:18:21.858698 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:21.858676    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c0d9f03-e1b9-42e7-a808-afa4de60bce5-maas-api-service-ca\") pod \"4c0d9f03-e1b9-42e7-a808-afa4de60bce5\" (UID: \"4c0d9f03-e1b9-42e7-a808-afa4de60bce5\") "
Jul 17 13:18:21.859027 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:21.859004    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0d9f03-e1b9-42e7-a808-afa4de60bce5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4c0d9f03-e1b9-42e7-a808-afa4de60bce5" (UID: "4c0d9f03-e1b9-42e7-a808-afa4de60bce5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:18:21.960022 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:21.959969    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c0d9f03-e1b9-42e7-a808-afa4de60bce5-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:18:22.665553 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:22.665529    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq"
Jul 17 13:18:22.665995 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:22.665521    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq" event={"ID":"4c0d9f03-e1b9-42e7-a808-afa4de60bce5","Type":"ContainerDied","Data":"5ab69851f7f71ce676132209f2f6e91faab58c23a93afcd0a497446659d1a4fb"}
Jul 17 13:18:22.665995 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:22.665690    2569 scope.go:117] "RemoveContainer" containerID="2fc93b294987fc894213cb030373a7652b46ee0e663c8e15c32f8759a5098394"
Jul 17 13:18:22.689764 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:22.689729    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq"]
Jul 17 13:18:22.697299 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:22.697276    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w8hkq"]
Jul 17 13:18:23.633489 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:23.633458    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c0d9f03-e1b9-42e7-a808-afa4de60bce5" path="/var/lib/kubelet/pods/4c0d9f03-e1b9-42e7-a808-afa4de60bce5/volumes"
Jul 17 13:18:23.861107 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:23.861077    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:18:30.391441 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391410    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"]
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391701    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c0d9f03-e1b9-42e7-a808-afa4de60bce5" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391712    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c0d9f03-e1b9-42e7-a808-afa4de60bce5" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391722    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67273611-969c-419b-8a2c-f1da39bb4679" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391728    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="67273611-969c-419b-8a2c-f1da39bb4679" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391735    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938ae4de-6532-4720-be35-6d1c8b21093f" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391741    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="938ae4de-6532-4720-be35-6d1c8b21093f" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391764    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c0d9f03-e1b9-42e7-a808-afa4de60bce5" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391772    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c0d9f03-e1b9-42e7-a808-afa4de60bce5" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391793    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938ae4de-6532-4720-be35-6d1c8b21093f" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391800    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="938ae4de-6532-4720-be35-6d1c8b21093f" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391807    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938ae4de-6532-4720-be35-6d1c8b21093f" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391827    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="938ae4de-6532-4720-be35-6d1c8b21093f" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391840    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67273611-969c-419b-8a2c-f1da39bb4679" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391845    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="67273611-969c-419b-8a2c-f1da39bb4679" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391898    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c0d9f03-e1b9-42e7-a808-afa4de60bce5" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391905    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="938ae4de-6532-4720-be35-6d1c8b21093f" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391913    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c0d9f03-e1b9-42e7-a808-afa4de60bce5" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391921    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="67273611-969c-419b-8a2c-f1da39bb4679" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391926    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="67273611-969c-419b-8a2c-f1da39bb4679" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391931    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="938ae4de-6532-4720-be35-6d1c8b21093f" containerName="revoke-keys"
Jul 17 13:18:30.391922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.391937    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="67273611-969c-419b-8a2c-f1da39bb4679" containerName="revoke-keys"
Jul 17 13:18:30.396063 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.396042    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"
Jul 17 13:18:30.399114 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.399090    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lwvzj\""
Jul 17 13:18:30.399952 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.399931    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 13:18:30.400086 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.399972    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cae85d5e-serving-cert\""
Jul 17 13:18:30.400215 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.400196    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:18:30.404871 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.404850    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"]
Jul 17 13:18:30.519855 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.519830    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26c12a0e-ecf4-4f42-b5b3-e450d2900252-maas-api-tls\") pod \"maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb\" (UID: \"26c12a0e-ecf4-4f42-b5b3-e450d2900252\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"
Jul 17 13:18:30.520006 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.519935    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c12a0e-ecf4-4f42-b5b3-e450d2900252-openshift-service-ca\") pod \"maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb\" (UID: \"26c12a0e-ecf4-4f42-b5b3-e450d2900252\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"
Jul 17 13:18:30.520006 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.519962    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t86vc\" (UniqueName: \"kubernetes.io/projected/26c12a0e-ecf4-4f42-b5b3-e450d2900252-kube-api-access-t86vc\") pod \"maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb\" (UID: \"26c12a0e-ecf4-4f42-b5b3-e450d2900252\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"
Jul 17 13:18:30.621186 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.621160    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c12a0e-ecf4-4f42-b5b3-e450d2900252-openshift-service-ca\") pod \"maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb\" (UID: \"26c12a0e-ecf4-4f42-b5b3-e450d2900252\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"
Jul 17 13:18:30.621312 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.621188    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t86vc\" (UniqueName: \"kubernetes.io/projected/26c12a0e-ecf4-4f42-b5b3-e450d2900252-kube-api-access-t86vc\") pod \"maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb\" (UID: \"26c12a0e-ecf4-4f42-b5b3-e450d2900252\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"
Jul 17 13:18:30.621312 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.621237    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26c12a0e-ecf4-4f42-b5b3-e450d2900252-maas-api-tls\") pod \"maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb\" (UID: \"26c12a0e-ecf4-4f42-b5b3-e450d2900252\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"
Jul 17 13:18:30.621429 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:18:30.621327    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-serving-cert: secret "maas-api-e2e-mt-cae85d5e-serving-cert" not found
Jul 17 13:18:30.621429 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:18:30.621376    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26c12a0e-ecf4-4f42-b5b3-e450d2900252-maas-api-tls podName:26c12a0e-ecf4-4f42-b5b3-e450d2900252 nodeName:}" failed. No retries permitted until 2026-07-17 13:18:31.121361762 +0000 UTC m=+3386.125408622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/26c12a0e-ecf4-4f42-b5b3-e450d2900252-maas-api-tls") pod "maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb" (UID: "26c12a0e-ecf4-4f42-b5b3-e450d2900252") : secret "maas-api-e2e-mt-cae85d5e-serving-cert" not found
Jul 17 13:18:30.621798 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.621780    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c12a0e-ecf4-4f42-b5b3-e450d2900252-openshift-service-ca\") pod \"maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb\" (UID: \"26c12a0e-ecf4-4f42-b5b3-e450d2900252\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"
Jul 17 13:18:30.635171 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:30.635141    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t86vc\" (UniqueName: \"kubernetes.io/projected/26c12a0e-ecf4-4f42-b5b3-e450d2900252-kube-api-access-t86vc\") pod \"maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb\" (UID: \"26c12a0e-ecf4-4f42-b5b3-e450d2900252\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"
Jul 17 13:18:31.125262 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:31.125229    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26c12a0e-ecf4-4f42-b5b3-e450d2900252-maas-api-tls\") pod \"maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb\" (UID: \"26c12a0e-ecf4-4f42-b5b3-e450d2900252\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"
Jul 17 13:18:31.127645 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:31.127616    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26c12a0e-ecf4-4f42-b5b3-e450d2900252-maas-api-tls\") pod \"maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb\" (UID: \"26c12a0e-ecf4-4f42-b5b3-e450d2900252\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"
Jul 17 13:18:31.307388 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:31.307358    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"
Jul 17 13:18:31.436580 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:31.436555    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"]
Jul 17 13:18:31.438722 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:18:31.438694    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c12a0e_ecf4_4f42_b5b3_e450d2900252.slice/crio-77e655163b64561fe95647d9044ed8ae51398ca3d8943a209d5a891cbbe6b2fe WatchSource:0}: Error finding container 77e655163b64561fe95647d9044ed8ae51398ca3d8943a209d5a891cbbe6b2fe: Status 404 returned error can't find the container with id 77e655163b64561fe95647d9044ed8ae51398ca3d8943a209d5a891cbbe6b2fe
Jul 17 13:18:31.440030 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:31.440012    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 13:18:31.705600 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:31.705539    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb" event={"ID":"26c12a0e-ecf4-4f42-b5b3-e450d2900252","Type":"ContainerStarted","Data":"77e655163b64561fe95647d9044ed8ae51398ca3d8943a209d5a891cbbe6b2fe"}
Jul 17 13:18:32.711797 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:32.711740    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb" event={"ID":"26c12a0e-ecf4-4f42-b5b3-e450d2900252","Type":"ContainerStarted","Data":"f4fb4f487f126dab0df3849d6f0f73b69cf5ffbe081e708118b75f4c305f36e9"}
Jul 17 13:18:32.712203 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:32.711831    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"
Jul 17 13:18:32.730795 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:32.730717    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb" podStartSLOduration=2.2588200609999998 podStartE2EDuration="2.730703349s" totalImagesPullingTime="471.883288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:18:32.729629024 +0000 UTC m=+3387.733675907" watchObservedRunningTime="2026-07-17 13:18:32.730703349 +0000 UTC m=+3387.734750227"
Jul 17 13:18:35.295827 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:35.295793    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:18:38.722174 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:38.722141    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"
Jul 17 13:18:41.924304 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:41.924262    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl"]
Jul 17 13:18:41.924724 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:41.924596    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67273611-969c-419b-8a2c-f1da39bb4679" containerName="revoke-keys"
Jul 17 13:18:41.924724 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:41.924608    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="67273611-969c-419b-8a2c-f1da39bb4679" containerName="revoke-keys"
Jul 17 13:18:41.924724 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:41.924636    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c0d9f03-e1b9-42e7-a808-afa4de60bce5" containerName="revoke-keys"
Jul 17 13:18:41.924724 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:41.924641    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c0d9f03-e1b9-42e7-a808-afa4de60bce5" containerName="revoke-keys"
Jul 17 13:18:41.924724 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:41.924695    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c0d9f03-e1b9-42e7-a808-afa4de60bce5" containerName="revoke-keys"
Jul 17 13:18:41.924724 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:41.924706    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="938ae4de-6532-4720-be35-6d1c8b21093f" containerName="revoke-keys"
Jul 17 13:18:41.929392 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:41.929356    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl"
Jul 17 13:18:41.932557 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:41.932522    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:18:41.935984 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:41.935945    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl"]
Jul 17 13:18:42.010661 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:42.010637    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/581313ce-1e8a-4cdd-8e64-e12bc5ce6e31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl\" (UID: \"581313ce-1e8a-4cdd-8e64-e12bc5ce6e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl"
Jul 17 13:18:42.111000 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:42.110971    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/581313ce-1e8a-4cdd-8e64-e12bc5ce6e31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl\" (UID: \"581313ce-1e8a-4cdd-8e64-e12bc5ce6e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl"
Jul 17 13:18:42.111586 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:42.111567    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/581313ce-1e8a-4cdd-8e64-e12bc5ce6e31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl\" (UID: \"581313ce-1e8a-4cdd-8e64-e12bc5ce6e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl"
Jul 17 13:18:42.244621 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:42.244602    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl"
Jul 17 13:18:42.376125 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:42.376105    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl"]
Jul 17 13:18:42.378781 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:18:42.378733    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581313ce_1e8a_4cdd_8e64_e12bc5ce6e31.slice/crio-3637d09fc89dfac559432f3161e9889fa97fadde58302593689be450d1b8f9c1 WatchSource:0}: Error finding container 3637d09fc89dfac559432f3161e9889fa97fadde58302593689be450d1b8f9c1: Status 404 returned error can't find the container with id 3637d09fc89dfac559432f3161e9889fa97fadde58302593689be450d1b8f9c1
Jul 17 13:18:42.758884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:42.758848    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl" event={"ID":"581313ce-1e8a-4cdd-8e64-e12bc5ce6e31","Type":"ContainerStarted","Data":"2a2b602aa097d6b4c264794ece503ea80dc80d6f9b5c2060acde99c4da095d4f"}
Jul 17 13:18:42.759014 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:42.758914    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl" event={"ID":"581313ce-1e8a-4cdd-8e64-e12bc5ce6e31","Type":"ContainerStarted","Data":"3637d09fc89dfac559432f3161e9889fa97fadde58302593689be450d1b8f9c1"}
Jul 17 13:18:42.775388 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:42.775347    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl" podStartSLOduration=1.775333325 podStartE2EDuration="1.775333325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:18:42.774895191 +0000 UTC m=+3397.778942075" watchObservedRunningTime="2026-07-17 13:18:42.775333325 +0000 UTC m=+3397.779380207"
Jul 17 13:18:43.763960 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:43.763923    2569 generic.go:356] "Generic (PLEG): container finished" podID="581313ce-1e8a-4cdd-8e64-e12bc5ce6e31" containerID="2a2b602aa097d6b4c264794ece503ea80dc80d6f9b5c2060acde99c4da095d4f" exitCode=0
Jul 17 13:18:43.764328 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:43.764006    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl" event={"ID":"581313ce-1e8a-4cdd-8e64-e12bc5ce6e31","Type":"ContainerDied","Data":"2a2b602aa097d6b4c264794ece503ea80dc80d6f9b5c2060acde99c4da095d4f"}
Jul 17 13:18:44.902040 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:44.902016    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl"
Jul 17 13:18:45.036350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:45.036294    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/581313ce-1e8a-4cdd-8e64-e12bc5ce6e31-maas-api-service-ca\") pod \"581313ce-1e8a-4cdd-8e64-e12bc5ce6e31\" (UID: \"581313ce-1e8a-4cdd-8e64-e12bc5ce6e31\") "
Jul 17 13:18:45.036623 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:45.036600    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/581313ce-1e8a-4cdd-8e64-e12bc5ce6e31-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "581313ce-1e8a-4cdd-8e64-e12bc5ce6e31" (UID: "581313ce-1e8a-4cdd-8e64-e12bc5ce6e31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:18:45.136817 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:45.136795    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/581313ce-1e8a-4cdd-8e64-e12bc5ce6e31-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:18:45.775293 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:45.775265    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl"
Jul 17 13:18:45.775443 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:45.775261    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl" event={"ID":"581313ce-1e8a-4cdd-8e64-e12bc5ce6e31","Type":"ContainerDied","Data":"3637d09fc89dfac559432f3161e9889fa97fadde58302593689be450d1b8f9c1"}
Jul 17 13:18:45.775443 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:45.775403    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3637d09fc89dfac559432f3161e9889fa97fadde58302593689be450d1b8f9c1"
Jul 17 13:18:51.013047 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:51.013008    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6"]
Jul 17 13:18:51.013486 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:51.013341    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="581313ce-1e8a-4cdd-8e64-e12bc5ce6e31" containerName="revoke-keys"
Jul 17 13:18:51.013486 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:51.013353    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="581313ce-1e8a-4cdd-8e64-e12bc5ce6e31" containerName="revoke-keys"
Jul 17 13:18:51.013486 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:51.013423    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="581313ce-1e8a-4cdd-8e64-e12bc5ce6e31" containerName="revoke-keys"
Jul 17 13:18:51.016332 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:51.016309    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6"
Jul 17 13:18:51.019876 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:51.019851    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:18:51.025208 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:51.025182    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6"]
Jul 17 13:18:51.184521 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:51.184492    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8b842d7-f7c5-46ab-bc88-1797f631df68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6\" (UID: \"e8b842d7-f7c5-46ab-bc88-1797f631df68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6"
Jul 17 13:18:51.285291 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:51.285203    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8b842d7-f7c5-46ab-bc88-1797f631df68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6\" (UID: \"e8b842d7-f7c5-46ab-bc88-1797f631df68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6"
Jul 17 13:18:51.285881 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:51.285854    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8b842d7-f7c5-46ab-bc88-1797f631df68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6\" (UID: \"e8b842d7-f7c5-46ab-bc88-1797f631df68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6"
Jul 17 13:18:51.328026 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:51.328000    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6"
Jul 17 13:18:51.455362 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:51.455338    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6"]
Jul 17 13:18:51.457713 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:18:51.457684    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b842d7_f7c5_46ab_bc88_1797f631df68.slice/crio-70689a97cb12e75ff287ddcc86dc693be350d5e43af5f57089cdecd231e82d8c WatchSource:0}: Error finding container 70689a97cb12e75ff287ddcc86dc693be350d5e43af5f57089cdecd231e82d8c: Status 404 returned error can't find the container with id 70689a97cb12e75ff287ddcc86dc693be350d5e43af5f57089cdecd231e82d8c
Jul 17 13:18:51.803778 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:51.803701    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6" event={"ID":"e8b842d7-f7c5-46ab-bc88-1797f631df68","Type":"ContainerStarted","Data":"ac5349acf52332072c56c894851602f1f9f930a2da1a90d9bd31cb2464bb581c"}
Jul 17 13:18:51.803884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:51.803769    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6" event={"ID":"e8b842d7-f7c5-46ab-bc88-1797f631df68","Type":"ContainerStarted","Data":"70689a97cb12e75ff287ddcc86dc693be350d5e43af5f57089cdecd231e82d8c"}
Jul 17 13:18:51.821546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:51.821508    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6" podStartSLOduration=1.821494236 podStartE2EDuration="1.821494236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:18:51.819304176 +0000 UTC m=+3406.823351060" watchObservedRunningTime="2026-07-17 13:18:51.821494236 +0000 UTC m=+3406.825541117"
Jul 17 13:18:52.502042 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.502009    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"]
Jul 17 13:18:52.502462 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.502251    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb" podUID="26c12a0e-ecf4-4f42-b5b3-e450d2900252" containerName="maas-api" containerID="cri-o://f4fb4f487f126dab0df3849d6f0f73b69cf5ffbe081e708118b75f4c305f36e9" gracePeriod=30
Jul 17 13:18:52.752866 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.752803    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"
Jul 17 13:18:52.809166 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.809135    2569 generic.go:356] "Generic (PLEG): container finished" podID="26c12a0e-ecf4-4f42-b5b3-e450d2900252" containerID="f4fb4f487f126dab0df3849d6f0f73b69cf5ffbe081e708118b75f4c305f36e9" exitCode=0
Jul 17 13:18:52.809166 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.809167    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"
Jul 17 13:18:52.809360 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.809207    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb" event={"ID":"26c12a0e-ecf4-4f42-b5b3-e450d2900252","Type":"ContainerDied","Data":"f4fb4f487f126dab0df3849d6f0f73b69cf5ffbe081e708118b75f4c305f36e9"}
Jul 17 13:18:52.809360 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.809281    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb" event={"ID":"26c12a0e-ecf4-4f42-b5b3-e450d2900252","Type":"ContainerDied","Data":"77e655163b64561fe95647d9044ed8ae51398ca3d8943a209d5a891cbbe6b2fe"}
Jul 17 13:18:52.809360 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.809315    2569 scope.go:117] "RemoveContainer" containerID="f4fb4f487f126dab0df3849d6f0f73b69cf5ffbe081e708118b75f4c305f36e9"
Jul 17 13:18:52.810621 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.810597    2569 generic.go:356] "Generic (PLEG): container finished" podID="e8b842d7-f7c5-46ab-bc88-1797f631df68" containerID="ac5349acf52332072c56c894851602f1f9f930a2da1a90d9bd31cb2464bb581c" exitCode=6
Jul 17 13:18:52.810761 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.810643    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6" event={"ID":"e8b842d7-f7c5-46ab-bc88-1797f631df68","Type":"ContainerDied","Data":"ac5349acf52332072c56c894851602f1f9f930a2da1a90d9bd31cb2464bb581c"}
Jul 17 13:18:52.810859 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.810845    2569 scope.go:117] "RemoveContainer" containerID="ac5349acf52332072c56c894851602f1f9f930a2da1a90d9bd31cb2464bb581c"
Jul 17 13:18:52.819781 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.819740    2569 scope.go:117] "RemoveContainer" containerID="f4fb4f487f126dab0df3849d6f0f73b69cf5ffbe081e708118b75f4c305f36e9"
Jul 17 13:18:52.820029 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:18:52.820011    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4fb4f487f126dab0df3849d6f0f73b69cf5ffbe081e708118b75f4c305f36e9\": container with ID starting with f4fb4f487f126dab0df3849d6f0f73b69cf5ffbe081e708118b75f4c305f36e9 not found: ID does not exist" containerID="f4fb4f487f126dab0df3849d6f0f73b69cf5ffbe081e708118b75f4c305f36e9"
Jul 17 13:18:52.820092 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.820036    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4fb4f487f126dab0df3849d6f0f73b69cf5ffbe081e708118b75f4c305f36e9"} err="failed to get container status \"f4fb4f487f126dab0df3849d6f0f73b69cf5ffbe081e708118b75f4c305f36e9\": rpc error: code = NotFound desc = could not find container \"f4fb4f487f126dab0df3849d6f0f73b69cf5ffbe081e708118b75f4c305f36e9\": container with ID starting with f4fb4f487f126dab0df3849d6f0f73b69cf5ffbe081e708118b75f4c305f36e9 not found: ID does not exist"
Jul 17 13:18:52.900320 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.900292    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26c12a0e-ecf4-4f42-b5b3-e450d2900252-maas-api-tls\") pod \"26c12a0e-ecf4-4f42-b5b3-e450d2900252\" (UID: \"26c12a0e-ecf4-4f42-b5b3-e450d2900252\") "
Jul 17 13:18:52.900420 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.900335    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t86vc\" (UniqueName: \"kubernetes.io/projected/26c12a0e-ecf4-4f42-b5b3-e450d2900252-kube-api-access-t86vc\") pod \"26c12a0e-ecf4-4f42-b5b3-e450d2900252\" (UID: \"26c12a0e-ecf4-4f42-b5b3-e450d2900252\") "
Jul 17 13:18:52.900420 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.900368    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c12a0e-ecf4-4f42-b5b3-e450d2900252-openshift-service-ca\") pod \"26c12a0e-ecf4-4f42-b5b3-e450d2900252\" (UID: \"26c12a0e-ecf4-4f42-b5b3-e450d2900252\") "
Jul 17 13:18:52.900677 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.900655    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c12a0e-ecf4-4f42-b5b3-e450d2900252-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "26c12a0e-ecf4-4f42-b5b3-e450d2900252" (UID: "26c12a0e-ecf4-4f42-b5b3-e450d2900252"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:18:52.902263 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.902225    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c12a0e-ecf4-4f42-b5b3-e450d2900252-kube-api-access-t86vc" (OuterVolumeSpecName: "kube-api-access-t86vc") pod "26c12a0e-ecf4-4f42-b5b3-e450d2900252" (UID: "26c12a0e-ecf4-4f42-b5b3-e450d2900252"). InnerVolumeSpecName "kube-api-access-t86vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:18:52.902263 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:52.902253    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c12a0e-ecf4-4f42-b5b3-e450d2900252-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "26c12a0e-ecf4-4f42-b5b3-e450d2900252" (UID: "26c12a0e-ecf4-4f42-b5b3-e450d2900252"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:18:53.001818 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:53.001791    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26c12a0e-ecf4-4f42-b5b3-e450d2900252-maas-api-tls\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:18:53.001818 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:53.001817    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t86vc\" (UniqueName: \"kubernetes.io/projected/26c12a0e-ecf4-4f42-b5b3-e450d2900252-kube-api-access-t86vc\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:18:53.001946 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:53.001827    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c12a0e-ecf4-4f42-b5b3-e450d2900252-openshift-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:18:53.130321 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:53.130298    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"]
Jul 17 13:18:53.132273 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:53.132251    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cae85d5e-6d96f84bbf-qxcsb"]
Jul 17 13:18:53.147441 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:18:53.147423    2569 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c12a0e_ecf4_4f42_b5b3_e450d2900252.slice\": RecentStats: unable to find data in memory cache]"
Jul 17 13:18:53.633952 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:53.633928    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26c12a0e-ecf4-4f42-b5b3-e450d2900252" path="/var/lib/kubelet/pods/26c12a0e-ecf4-4f42-b5b3-e450d2900252/volumes"
Jul 17 13:18:53.818220 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:53.818192    2569 generic.go:356] "Generic (PLEG): container finished" podID="e8b842d7-f7c5-46ab-bc88-1797f631df68" containerID="e740171ee3f6501b5aa92c25c64a494daab9ec5fc6f78aca342ae260675ce32e" exitCode=6
Jul 17 13:18:53.818369 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:53.818286    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6" event={"ID":"e8b842d7-f7c5-46ab-bc88-1797f631df68","Type":"ContainerDied","Data":"e740171ee3f6501b5aa92c25c64a494daab9ec5fc6f78aca342ae260675ce32e"}
Jul 17 13:18:53.818438 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:53.818369    2569 scope.go:117] "RemoveContainer" containerID="ac5349acf52332072c56c894851602f1f9f930a2da1a90d9bd31cb2464bb581c"
Jul 17 13:18:53.818648 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:53.818626    2569 scope.go:117] "RemoveContainer" containerID="e740171ee3f6501b5aa92c25c64a494daab9ec5fc6f78aca342ae260675ce32e"
Jul 17 13:18:53.818894 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:18:53.818870    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6_odh-ai-gateway-infra(e8b842d7-f7c5-46ab-bc88-1797f631df68)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6" podUID="e8b842d7-f7c5-46ab-bc88-1797f631df68"
Jul 17 13:18:54.825522 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:54.825492    2569 scope.go:117] "RemoveContainer" containerID="e740171ee3f6501b5aa92c25c64a494daab9ec5fc6f78aca342ae260675ce32e"
Jul 17 13:18:54.825940 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:18:54.825676    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6_odh-ai-gateway-infra(e8b842d7-f7c5-46ab-bc88-1797f631df68)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6" podUID="e8b842d7-f7c5-46ab-bc88-1797f631df68"
Jul 17 13:18:55.264699 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:55.264663    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:18:58.494248 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:58.494142    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w"]
Jul 17 13:18:58.494771 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:58.494666    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26c12a0e-ecf4-4f42-b5b3-e450d2900252" containerName="maas-api"
Jul 17 13:18:58.494771 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:58.494688    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="26c12a0e-ecf4-4f42-b5b3-e450d2900252" containerName="maas-api"
Jul 17 13:18:58.494909 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:58.494843    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="26c12a0e-ecf4-4f42-b5b3-e450d2900252" containerName="maas-api"
Jul 17 13:18:58.499181 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:58.499154    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w"
Jul 17 13:18:58.508413 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:58.508379    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w"]
Jul 17 13:18:58.642995 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:58.642966    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e0cbe5-f20e-42e7-8e0d-68db8300d313-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w\" (UID: \"c8e0cbe5-f20e-42e7-8e0d-68db8300d313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w"
Jul 17 13:18:58.744340 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:58.744277    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e0cbe5-f20e-42e7-8e0d-68db8300d313-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w\" (UID: \"c8e0cbe5-f20e-42e7-8e0d-68db8300d313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w"
Jul 17 13:18:58.744948 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:58.744923    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e0cbe5-f20e-42e7-8e0d-68db8300d313-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w\" (UID: \"c8e0cbe5-f20e-42e7-8e0d-68db8300d313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w"
Jul 17 13:18:58.811416 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:58.811392    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w"
Jul 17 13:18:58.941919 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:58.941851    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w"]
Jul 17 13:18:58.944262 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:18:58.944235    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e0cbe5_f20e_42e7_8e0d_68db8300d313.slice/crio-84ae6bbc18f4c299f967850abed5b46c478d8a7c0d86f2d93e0c55bb029bc77b WatchSource:0}: Error finding container 84ae6bbc18f4c299f967850abed5b46c478d8a7c0d86f2d93e0c55bb029bc77b: Status 404 returned error can't find the container with id 84ae6bbc18f4c299f967850abed5b46c478d8a7c0d86f2d93e0c55bb029bc77b
Jul 17 13:18:59.852527 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:59.852447    2569 generic.go:356] "Generic (PLEG): container finished" podID="c8e0cbe5-f20e-42e7-8e0d-68db8300d313" containerID="e1b3465662a886d593c76052bf024e0db22a51931e3757179fc09edf5ecfd045" exitCode=6
Jul 17 13:18:59.852982 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:59.852538    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w" event={"ID":"c8e0cbe5-f20e-42e7-8e0d-68db8300d313","Type":"ContainerDied","Data":"e1b3465662a886d593c76052bf024e0db22a51931e3757179fc09edf5ecfd045"}
Jul 17 13:18:59.852982 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:59.852587    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w" event={"ID":"c8e0cbe5-f20e-42e7-8e0d-68db8300d313","Type":"ContainerStarted","Data":"84ae6bbc18f4c299f967850abed5b46c478d8a7c0d86f2d93e0c55bb029bc77b"}
Jul 17 13:18:59.852982 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:18:59.852711    2569 scope.go:117] "RemoveContainer" containerID="e1b3465662a886d593c76052bf024e0db22a51931e3757179fc09edf5ecfd045"
Jul 17 13:19:00.859430 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:00.859364    2569 generic.go:356] "Generic (PLEG): container finished" podID="c8e0cbe5-f20e-42e7-8e0d-68db8300d313" containerID="de361d16ef39c628db337a20dd71ec2dc91a162462749a26dcf118f541d8ee49" exitCode=6
Jul 17 13:19:00.859430 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:00.859424    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w" event={"ID":"c8e0cbe5-f20e-42e7-8e0d-68db8300d313","Type":"ContainerDied","Data":"de361d16ef39c628db337a20dd71ec2dc91a162462749a26dcf118f541d8ee49"}
Jul 17 13:19:00.859832 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:00.859471    2569 scope.go:117] "RemoveContainer" containerID="e1b3465662a886d593c76052bf024e0db22a51931e3757179fc09edf5ecfd045"
Jul 17 13:19:00.859832 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:00.859605    2569 scope.go:117] "RemoveContainer" containerID="de361d16ef39c628db337a20dd71ec2dc91a162462749a26dcf118f541d8ee49"
Jul 17 13:19:00.859832 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:19:00.859798    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w_odh-ai-gateway-infra(c8e0cbe5-f20e-42e7-8e0d-68db8300d313)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w" podUID="c8e0cbe5-f20e-42e7-8e0d-68db8300d313"
Jul 17 13:19:01.865982 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:01.865952    2569 scope.go:117] "RemoveContainer" containerID="de361d16ef39c628db337a20dd71ec2dc91a162462749a26dcf118f541d8ee49"
Jul 17 13:19:01.866375 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:19:01.866120    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w_odh-ai-gateway-infra(c8e0cbe5-f20e-42e7-8e0d-68db8300d313)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w" podUID="c8e0cbe5-f20e-42e7-8e0d-68db8300d313"
Jul 17 13:19:02.202014 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:02.201941    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s"]
Jul 17 13:19:02.206569 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:02.206551    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s"
Jul 17 13:19:02.226151 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:02.226126    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s"]
Jul 17 13:19:02.369344 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:02.369318    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30772e0e-d6a1-447c-b959-dc6723c1bd04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s\" (UID: \"30772e0e-d6a1-447c-b959-dc6723c1bd04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s"
Jul 17 13:19:02.469697 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:02.469635    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30772e0e-d6a1-447c-b959-dc6723c1bd04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s\" (UID: \"30772e0e-d6a1-447c-b959-dc6723c1bd04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s"
Jul 17 13:19:02.470219 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:02.470199    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30772e0e-d6a1-447c-b959-dc6723c1bd04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s\" (UID: \"30772e0e-d6a1-447c-b959-dc6723c1bd04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s"
Jul 17 13:19:02.516436 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:02.516413    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s"
Jul 17 13:19:02.638335 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:02.638283    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s"]
Jul 17 13:19:02.640367 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:19:02.640326    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30772e0e_d6a1_447c_b959_dc6723c1bd04.slice/crio-0b3cc17ec428d6d85a658c504047c88af04613e8776f6370510f097217d299e3 WatchSource:0}: Error finding container 0b3cc17ec428d6d85a658c504047c88af04613e8776f6370510f097217d299e3: Status 404 returned error can't find the container with id 0b3cc17ec428d6d85a658c504047c88af04613e8776f6370510f097217d299e3
Jul 17 13:19:02.871215 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:02.871187    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s" event={"ID":"30772e0e-d6a1-447c-b959-dc6723c1bd04","Type":"ContainerStarted","Data":"c49a4644d07387a016fcc2113835a6566c1a6fa36c3e79f23caf86358046fb60"}
Jul 17 13:19:02.871510 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:02.871233    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s" event={"ID":"30772e0e-d6a1-447c-b959-dc6723c1bd04","Type":"ContainerStarted","Data":"0b3cc17ec428d6d85a658c504047c88af04613e8776f6370510f097217d299e3"}
Jul 17 13:19:02.887495 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:02.887427    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s" podStartSLOduration=0.887407118 podStartE2EDuration="887.407118ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:19:02.886665446 +0000 UTC m=+3417.890712311" watchObservedRunningTime="2026-07-17 13:19:02.887407118 +0000 UTC m=+3417.891453982"
Jul 17 13:19:03.877236 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:03.877207    2569 generic.go:356] "Generic (PLEG): container finished" podID="30772e0e-d6a1-447c-b959-dc6723c1bd04" containerID="c49a4644d07387a016fcc2113835a6566c1a6fa36c3e79f23caf86358046fb60" exitCode=6
Jul 17 13:19:03.877646 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:03.877289    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s" event={"ID":"30772e0e-d6a1-447c-b959-dc6723c1bd04","Type":"ContainerDied","Data":"c49a4644d07387a016fcc2113835a6566c1a6fa36c3e79f23caf86358046fb60"}
Jul 17 13:19:03.877646 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:03.877466    2569 scope.go:117] "RemoveContainer" containerID="c49a4644d07387a016fcc2113835a6566c1a6fa36c3e79f23caf86358046fb60"
Jul 17 13:19:04.883846 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:04.883816    2569 generic.go:356] "Generic (PLEG): container finished" podID="30772e0e-d6a1-447c-b959-dc6723c1bd04" containerID="ad782d466ef2d2433051fd41b9f0204a0da783727a7cd518f82224442ca4c7bd" exitCode=6
Jul 17 13:19:04.884225 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:04.883885    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s" event={"ID":"30772e0e-d6a1-447c-b959-dc6723c1bd04","Type":"ContainerDied","Data":"ad782d466ef2d2433051fd41b9f0204a0da783727a7cd518f82224442ca4c7bd"}
Jul 17 13:19:04.884225 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:04.883945    2569 scope.go:117] "RemoveContainer" containerID="c49a4644d07387a016fcc2113835a6566c1a6fa36c3e79f23caf86358046fb60"
Jul 17 13:19:04.884225 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:04.884106    2569 scope.go:117] "RemoveContainer" containerID="ad782d466ef2d2433051fd41b9f0204a0da783727a7cd518f82224442ca4c7bd"
Jul 17 13:19:04.884355 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:19:04.884288    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s_odh-ai-gateway-infra(30772e0e-d6a1-447c-b959-dc6723c1bd04)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s" podUID="30772e0e-d6a1-447c-b959-dc6723c1bd04"
Jul 17 13:19:05.889864 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:05.889838    2569 scope.go:117] "RemoveContainer" containerID="ad782d466ef2d2433051fd41b9f0204a0da783727a7cd518f82224442ca4c7bd"
Jul 17 13:19:05.890220 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:19:05.890021    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s_odh-ai-gateway-infra(30772e0e-d6a1-447c-b959-dc6723c1bd04)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s" podUID="30772e0e-d6a1-447c-b959-dc6723c1bd04"
Jul 17 13:19:08.629127 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:08.629096    2569 scope.go:117] "RemoveContainer" containerID="e740171ee3f6501b5aa92c25c64a494daab9ec5fc6f78aca342ae260675ce32e"
Jul 17 13:19:09.910502 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:09.910468    2569 generic.go:356] "Generic (PLEG): container finished" podID="e8b842d7-f7c5-46ab-bc88-1797f631df68" containerID="7580ffe9fdf6f86f4d01ce44bac85547f68339d0f9f7f61171b7c25dbcc8fc31" exitCode=6
Jul 17 13:19:09.910970 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:09.910544    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6" event={"ID":"e8b842d7-f7c5-46ab-bc88-1797f631df68","Type":"ContainerDied","Data":"7580ffe9fdf6f86f4d01ce44bac85547f68339d0f9f7f61171b7c25dbcc8fc31"}
Jul 17 13:19:09.910970 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:09.910626    2569 scope.go:117] "RemoveContainer" containerID="e740171ee3f6501b5aa92c25c64a494daab9ec5fc6f78aca342ae260675ce32e"
Jul 17 13:19:09.910970 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:09.910920    2569 scope.go:117] "RemoveContainer" containerID="7580ffe9fdf6f86f4d01ce44bac85547f68339d0f9f7f61171b7c25dbcc8fc31"
Jul 17 13:19:09.911159 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:19:09.911137    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6_odh-ai-gateway-infra(e8b842d7-f7c5-46ab-bc88-1797f631df68)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6" podUID="e8b842d7-f7c5-46ab-bc88-1797f631df68"
Jul 17 13:19:10.953619 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:10.953583    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6"]
Jul 17 13:19:11.081717 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:11.081692    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6"
Jul 17 13:19:11.138507 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:11.138481    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8b842d7-f7c5-46ab-bc88-1797f631df68-maas-api-service-ca\") pod \"e8b842d7-f7c5-46ab-bc88-1797f631df68\" (UID: \"e8b842d7-f7c5-46ab-bc88-1797f631df68\") "
Jul 17 13:19:11.138797 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:11.138775    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b842d7-f7c5-46ab-bc88-1797f631df68-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e8b842d7-f7c5-46ab-bc88-1797f631df68" (UID: "e8b842d7-f7c5-46ab-bc88-1797f631df68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:19:11.238984 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:11.238963    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8b842d7-f7c5-46ab-bc88-1797f631df68-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:19:11.923036 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:11.922975    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6" event={"ID":"e8b842d7-f7c5-46ab-bc88-1797f631df68","Type":"ContainerDied","Data":"70689a97cb12e75ff287ddcc86dc693be350d5e43af5f57089cdecd231e82d8c"}
Jul 17 13:19:11.923036 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:11.923029    2569 scope.go:117] "RemoveContainer" containerID="7580ffe9fdf6f86f4d01ce44bac85547f68339d0f9f7f61171b7c25dbcc8fc31"
Jul 17 13:19:11.923205 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:11.922982    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6"
Jul 17 13:19:11.939963 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:11.939940    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6"]
Jul 17 13:19:11.943130 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:11.943109    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-j79p6"]
Jul 17 13:19:13.629074 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:13.629040    2569 scope.go:117] "RemoveContainer" containerID="de361d16ef39c628db337a20dd71ec2dc91a162462749a26dcf118f541d8ee49"
Jul 17 13:19:13.636148 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:13.636125    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8b842d7-f7c5-46ab-bc88-1797f631df68" path="/var/lib/kubelet/pods/e8b842d7-f7c5-46ab-bc88-1797f631df68/volumes"
Jul 17 13:19:14.942289 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:14.942255    2569 generic.go:356] "Generic (PLEG): container finished" podID="c8e0cbe5-f20e-42e7-8e0d-68db8300d313" containerID="fbaa0a8e28b8e4f751bd2911633ad88b93e8c9b150a102ea37e35baa20182392" exitCode=6
Jul 17 13:19:14.942710 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:14.942324    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w" event={"ID":"c8e0cbe5-f20e-42e7-8e0d-68db8300d313","Type":"ContainerDied","Data":"fbaa0a8e28b8e4f751bd2911633ad88b93e8c9b150a102ea37e35baa20182392"}
Jul 17 13:19:14.942710 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:14.942383    2569 scope.go:117] "RemoveContainer" containerID="de361d16ef39c628db337a20dd71ec2dc91a162462749a26dcf118f541d8ee49"
Jul 17 13:19:14.942710 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:14.942634    2569 scope.go:117] "RemoveContainer" containerID="fbaa0a8e28b8e4f751bd2911633ad88b93e8c9b150a102ea37e35baa20182392"
Jul 17 13:19:14.942895 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:19:14.942851    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w_odh-ai-gateway-infra(c8e0cbe5-f20e-42e7-8e0d-68db8300d313)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w" podUID="c8e0cbe5-f20e-42e7-8e0d-68db8300d313"
Jul 17 13:19:15.974399 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:15.974368    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w"]
Jul 17 13:19:16.109583 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:16.109555    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w"
Jul 17 13:19:16.173438 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:16.173410    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e0cbe5-f20e-42e7-8e0d-68db8300d313-maas-api-service-ca\") pod \"c8e0cbe5-f20e-42e7-8e0d-68db8300d313\" (UID: \"c8e0cbe5-f20e-42e7-8e0d-68db8300d313\") "
Jul 17 13:19:16.173733 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:16.173711    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e0cbe5-f20e-42e7-8e0d-68db8300d313-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c8e0cbe5-f20e-42e7-8e0d-68db8300d313" (UID: "c8e0cbe5-f20e-42e7-8e0d-68db8300d313"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:19:16.274705 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:16.274679    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8e0cbe5-f20e-42e7-8e0d-68db8300d313-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:19:16.954358 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:16.954324    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w"
Jul 17 13:19:16.954358 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:16.954351    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w" event={"ID":"c8e0cbe5-f20e-42e7-8e0d-68db8300d313","Type":"ContainerDied","Data":"84ae6bbc18f4c299f967850abed5b46c478d8a7c0d86f2d93e0c55bb029bc77b"}
Jul 17 13:19:16.954586 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:16.954415    2569 scope.go:117] "RemoveContainer" containerID="fbaa0a8e28b8e4f751bd2911633ad88b93e8c9b150a102ea37e35baa20182392"
Jul 17 13:19:16.976279 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:16.976254    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w"]
Jul 17 13:19:16.979366 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:16.979345    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9rl8w"]
Jul 17 13:19:17.634457 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:17.634424    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8e0cbe5-f20e-42e7-8e0d-68db8300d313" path="/var/lib/kubelet/pods/c8e0cbe5-f20e-42e7-8e0d-68db8300d313/volumes"
Jul 17 13:19:20.629273 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:20.629245    2569 scope.go:117] "RemoveContainer" containerID="ad782d466ef2d2433051fd41b9f0204a0da783727a7cd518f82224442ca4c7bd"
Jul 17 13:19:20.977420 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:20.977396    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s" event={"ID":"30772e0e-d6a1-447c-b959-dc6723c1bd04","Type":"ContainerStarted","Data":"4e7bbe4447a1e9ca3ba704e2469df1f45c3216f6f700db10f8d4aa0c876227e5"}
Jul 17 13:19:21.983670 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:21.983637    2569 generic.go:356] "Generic (PLEG): container finished" podID="30772e0e-d6a1-447c-b959-dc6723c1bd04" containerID="4e7bbe4447a1e9ca3ba704e2469df1f45c3216f6f700db10f8d4aa0c876227e5" exitCode=6
Jul 17 13:19:21.984123 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:21.983711    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s" event={"ID":"30772e0e-d6a1-447c-b959-dc6723c1bd04","Type":"ContainerDied","Data":"4e7bbe4447a1e9ca3ba704e2469df1f45c3216f6f700db10f8d4aa0c876227e5"}
Jul 17 13:19:21.984123 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:21.983799    2569 scope.go:117] "RemoveContainer" containerID="ad782d466ef2d2433051fd41b9f0204a0da783727a7cd518f82224442ca4c7bd"
Jul 17 13:19:21.984123 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:21.984109    2569 scope.go:117] "RemoveContainer" containerID="4e7bbe4447a1e9ca3ba704e2469df1f45c3216f6f700db10f8d4aa0c876227e5"
Jul 17 13:19:21.984361 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:19:21.984328    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s_odh-ai-gateway-infra(30772e0e-d6a1-447c-b959-dc6723c1bd04)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s" podUID="30772e0e-d6a1-447c-b959-dc6723c1bd04"
Jul 17 13:19:22.008099 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:22.008069    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s"]
Jul 17 13:19:23.126552 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:23.126527    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s"
Jul 17 13:19:23.226966 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:23.226943    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30772e0e-d6a1-447c-b959-dc6723c1bd04-maas-api-service-ca\") pod \"30772e0e-d6a1-447c-b959-dc6723c1bd04\" (UID: \"30772e0e-d6a1-447c-b959-dc6723c1bd04\") "
Jul 17 13:19:23.227360 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:23.227333    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30772e0e-d6a1-447c-b959-dc6723c1bd04-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "30772e0e-d6a1-447c-b959-dc6723c1bd04" (UID: "30772e0e-d6a1-447c-b959-dc6723c1bd04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:19:23.327662 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:23.327608    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30772e0e-d6a1-447c-b959-dc6723c1bd04-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:19:23.996863 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:23.996821    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s" event={"ID":"30772e0e-d6a1-447c-b959-dc6723c1bd04","Type":"ContainerDied","Data":"0b3cc17ec428d6d85a658c504047c88af04613e8776f6370510f097217d299e3"}
Jul 17 13:19:23.996998 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:23.996890    2569 scope.go:117] "RemoveContainer" containerID="4e7bbe4447a1e9ca3ba704e2469df1f45c3216f6f700db10f8d4aa0c876227e5"
Jul 17 13:19:23.996998 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:23.996832    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s"
Jul 17 13:19:24.024364 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:24.024339    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s"]
Jul 17 13:19:24.032937 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:24.032916    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cxp9s"]
Jul 17 13:19:25.634906 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:25.634860    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30772e0e-d6a1-447c-b959-dc6723c1bd04" path="/var/lib/kubelet/pods/30772e0e-d6a1-447c-b959-dc6723c1bd04/volumes"
Jul 17 13:19:31.535982 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.535949    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"]
Jul 17 13:19:31.536348 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536273    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e0cbe5-f20e-42e7-8e0d-68db8300d313" containerName="revoke-keys"
Jul 17 13:19:31.536348 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536289    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e0cbe5-f20e-42e7-8e0d-68db8300d313" containerName="revoke-keys"
Jul 17 13:19:31.536348 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536297    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e0cbe5-f20e-42e7-8e0d-68db8300d313" containerName="revoke-keys"
Jul 17 13:19:31.536348 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536303    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e0cbe5-f20e-42e7-8e0d-68db8300d313" containerName="revoke-keys"
Jul 17 13:19:31.536348 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536316    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8b842d7-f7c5-46ab-bc88-1797f631df68" containerName="revoke-keys"
Jul 17 13:19:31.536348 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536321    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8b842d7-f7c5-46ab-bc88-1797f631df68" containerName="revoke-keys"
Jul 17 13:19:31.536348 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536327    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8b842d7-f7c5-46ab-bc88-1797f631df68" containerName="revoke-keys"
Jul 17 13:19:31.536348 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536334    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8b842d7-f7c5-46ab-bc88-1797f631df68" containerName="revoke-keys"
Jul 17 13:19:31.536348 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536340    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30772e0e-d6a1-447c-b959-dc6723c1bd04" containerName="revoke-keys"
Jul 17 13:19:31.536348 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536346    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="30772e0e-d6a1-447c-b959-dc6723c1bd04" containerName="revoke-keys"
Jul 17 13:19:31.536663 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536354    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30772e0e-d6a1-447c-b959-dc6723c1bd04" containerName="revoke-keys"
Jul 17 13:19:31.536663 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536362    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="30772e0e-d6a1-447c-b959-dc6723c1bd04" containerName="revoke-keys"
Jul 17 13:19:31.536663 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536372    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e0cbe5-f20e-42e7-8e0d-68db8300d313" containerName="revoke-keys"
Jul 17 13:19:31.536663 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536377    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e0cbe5-f20e-42e7-8e0d-68db8300d313" containerName="revoke-keys"
Jul 17 13:19:31.536663 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536438    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8b842d7-f7c5-46ab-bc88-1797f631df68" containerName="revoke-keys"
Jul 17 13:19:31.536663 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536449    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8b842d7-f7c5-46ab-bc88-1797f631df68" containerName="revoke-keys"
Jul 17 13:19:31.536663 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536456    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8e0cbe5-f20e-42e7-8e0d-68db8300d313" containerName="revoke-keys"
Jul 17 13:19:31.536663 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536462    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="30772e0e-d6a1-447c-b959-dc6723c1bd04" containerName="revoke-keys"
Jul 17 13:19:31.536663 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536466    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="30772e0e-d6a1-447c-b959-dc6723c1bd04" containerName="revoke-keys"
Jul 17 13:19:31.536663 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536471    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="30772e0e-d6a1-447c-b959-dc6723c1bd04" containerName="revoke-keys"
Jul 17 13:19:31.536663 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.536476    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8e0cbe5-f20e-42e7-8e0d-68db8300d313" containerName="revoke-keys"
Jul 17 13:19:31.540883 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.540864    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"
Jul 17 13:19:31.544451 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.544430    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 13:19:31.545512 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.545494    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6ad1d9a1-serving-cert\""
Jul 17 13:19:31.545591 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.545526    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:19:31.545591 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.545501    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lwvzj\""
Jul 17 13:19:31.559123 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.559101    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"]
Jul 17 13:19:31.591422 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.591402    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd-maas-api-tls\") pod \"maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9\" (UID: \"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"
Jul 17 13:19:31.591518 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.591443    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm59b\" (UniqueName: \"kubernetes.io/projected/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd-kube-api-access-fm59b\") pod \"maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9\" (UID: \"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"
Jul 17 13:19:31.591518 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.591482    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd-openshift-service-ca\") pod \"maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9\" (UID: \"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"
Jul 17 13:19:31.692595 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.692566    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd-maas-api-tls\") pod \"maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9\" (UID: \"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"
Jul 17 13:19:31.692716 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.692628    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm59b\" (UniqueName: \"kubernetes.io/projected/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd-kube-api-access-fm59b\") pod \"maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9\" (UID: \"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"
Jul 17 13:19:31.692716 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.692678    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd-openshift-service-ca\") pod \"maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9\" (UID: \"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"
Jul 17 13:19:31.693323 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.693300    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd-openshift-service-ca\") pod \"maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9\" (UID: \"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"
Jul 17 13:19:31.695037 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.695017    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd-maas-api-tls\") pod \"maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9\" (UID: \"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"
Jul 17 13:19:31.703212 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.703184    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm59b\" (UniqueName: \"kubernetes.io/projected/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd-kube-api-access-fm59b\") pod \"maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9\" (UID: \"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"
Jul 17 13:19:31.850655 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.850601    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"
Jul 17 13:19:31.979247 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:31.979224    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"]
Jul 17 13:19:31.980729 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:19:31.980703    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e7ed7a_cc4b_4b71_a3c6_1ad4e1b8e6fd.slice/crio-a717950ee557627fc387ad0465bea190b4c288e6dfbf7059ed9f00bc9b58cb51 WatchSource:0}: Error finding container a717950ee557627fc387ad0465bea190b4c288e6dfbf7059ed9f00bc9b58cb51: Status 404 returned error can't find the container with id a717950ee557627fc387ad0465bea190b4c288e6dfbf7059ed9f00bc9b58cb51
Jul 17 13:19:32.039582 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:32.039551    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9" event={"ID":"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd","Type":"ContainerStarted","Data":"a717950ee557627fc387ad0465bea190b4c288e6dfbf7059ed9f00bc9b58cb51"}
Jul 17 13:19:33.045903 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:33.045870    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9" event={"ID":"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd","Type":"ContainerStarted","Data":"1f276333ba7dbe86b40167177b8bddc6e5b6a4d8c9ad910152b8db447bde3842"}
Jul 17 13:19:33.046277 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:33.045993    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"
Jul 17 13:19:33.077204 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:33.077159    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9" podStartSLOduration=1.6981847810000001 podStartE2EDuration="2.077147027s" totalImagesPullingTime="378.962246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:19:33.075170207 +0000 UTC m=+3448.079217090" watchObservedRunningTime="2026-07-17 13:19:33.077147027 +0000 UTC m=+3448.081193908"
Jul 17 13:19:36.767440 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:36.767408    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:19:39.056997 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:39.056959    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"
Jul 17 13:19:39.864592 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:39.864556    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t"]
Jul 17 13:19:39.864930 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:39.864914    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8b842d7-f7c5-46ab-bc88-1797f631df68" containerName="revoke-keys"
Jul 17 13:19:39.864930 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:39.864932    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8b842d7-f7c5-46ab-bc88-1797f631df68" containerName="revoke-keys"
Jul 17 13:19:39.865025 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:39.864942    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30772e0e-d6a1-447c-b959-dc6723c1bd04" containerName="revoke-keys"
Jul 17 13:19:39.865025 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:39.864949    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="30772e0e-d6a1-447c-b959-dc6723c1bd04" containerName="revoke-keys"
Jul 17 13:19:39.865025 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:39.865024    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8b842d7-f7c5-46ab-bc88-1797f631df68" containerName="revoke-keys"
Jul 17 13:19:39.865127 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:39.865037    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8e0cbe5-f20e-42e7-8e0d-68db8300d313" containerName="revoke-keys"
Jul 17 13:19:39.868655 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:39.868634    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t"
Jul 17 13:19:39.871392 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:39.871368    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:19:39.875245 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:39.875219    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t"]
Jul 17 13:19:39.953097 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:39.953070    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t\" (UID: \"05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t"
Jul 17 13:19:40.053417 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:40.053388    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t\" (UID: \"05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t"
Jul 17 13:19:40.053956 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:40.053935    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t\" (UID: \"05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t"
Jul 17 13:19:40.181967 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:40.181909    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t"
Jul 17 13:19:40.314898 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:40.314873    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t"]
Jul 17 13:19:40.316782 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:19:40.316740    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05bb3cca_b8a4_4a3d_bb4a_9b51dd13eec8.slice/crio-a74396677da74d439c961629629f25d20ed80275b6bfef1a0b35370ff9d08fd1 WatchSource:0}: Error finding container a74396677da74d439c961629629f25d20ed80275b6bfef1a0b35370ff9d08fd1: Status 404 returned error can't find the container with id a74396677da74d439c961629629f25d20ed80275b6bfef1a0b35370ff9d08fd1
Jul 17 13:19:41.084978 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:41.084952    2569 generic.go:356] "Generic (PLEG): container finished" podID="05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8" containerID="c6e208420257ca5c6493d938ca94a9c65c1428daf9baae6c58ce9599794b41be" exitCode=6
Jul 17 13:19:41.085105 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:41.085040    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t" event={"ID":"05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8","Type":"ContainerDied","Data":"c6e208420257ca5c6493d938ca94a9c65c1428daf9baae6c58ce9599794b41be"}
Jul 17 13:19:41.085105 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:41.085090    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t" event={"ID":"05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8","Type":"ContainerStarted","Data":"a74396677da74d439c961629629f25d20ed80275b6bfef1a0b35370ff9d08fd1"}
Jul 17 13:19:41.085194 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:41.085162    2569 scope.go:117] "RemoveContainer" containerID="c6e208420257ca5c6493d938ca94a9c65c1428daf9baae6c58ce9599794b41be"
Jul 17 13:19:42.091715 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:42.091686    2569 generic.go:356] "Generic (PLEG): container finished" podID="05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8" containerID="cc2df862e47f35b1130d7da0dc707b33bbd8390601fcc7d8ef3d18cb5d07d8de" exitCode=6
Jul 17 13:19:42.092083 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:42.091744    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t" event={"ID":"05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8","Type":"ContainerDied","Data":"cc2df862e47f35b1130d7da0dc707b33bbd8390601fcc7d8ef3d18cb5d07d8de"}
Jul 17 13:19:42.092083 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:42.091813    2569 scope.go:117] "RemoveContainer" containerID="c6e208420257ca5c6493d938ca94a9c65c1428daf9baae6c58ce9599794b41be"
Jul 17 13:19:42.092083 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:42.091953    2569 scope.go:117] "RemoveContainer" containerID="cc2df862e47f35b1130d7da0dc707b33bbd8390601fcc7d8ef3d18cb5d07d8de"
Jul 17 13:19:42.092232 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:19:42.092142    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t_odh-ai-gateway-infra(05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t" podUID="05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8"
Jul 17 13:19:43.099019 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:43.098988    2569 scope.go:117] "RemoveContainer" containerID="cc2df862e47f35b1130d7da0dc707b33bbd8390601fcc7d8ef3d18cb5d07d8de"
Jul 17 13:19:43.099387 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:19:43.099177    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t_odh-ai-gateway-infra(05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t" podUID="05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8"
Jul 17 13:19:51.078999 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:51.078963    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr"]
Jul 17 13:19:51.082362 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:51.082342    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr"
Jul 17 13:19:51.096044 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:51.095007    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr"]
Jul 17 13:19:51.141971 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:51.141942    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da58a42-233c-4f0c-a2ca-9415dee1b2f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr\" (UID: \"6da58a42-233c-4f0c-a2ca-9415dee1b2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr"
Jul 17 13:19:51.243062 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:51.243028    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da58a42-233c-4f0c-a2ca-9415dee1b2f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr\" (UID: \"6da58a42-233c-4f0c-a2ca-9415dee1b2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr"
Jul 17 13:19:51.243683 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:51.243658    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da58a42-233c-4f0c-a2ca-9415dee1b2f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr\" (UID: \"6da58a42-233c-4f0c-a2ca-9415dee1b2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr"
Jul 17 13:19:51.393131 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:51.393065    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr"
Jul 17 13:19:51.522529 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:51.522471    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr"]
Jul 17 13:19:51.524651 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:19:51.524622    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da58a42_233c_4f0c_a2ca_9415dee1b2f3.slice/crio-50c655b992d7171698512defea046dc582c11ec48f28fa7b5927e9592e4f3eeb WatchSource:0}: Error finding container 50c655b992d7171698512defea046dc582c11ec48f28fa7b5927e9592e4f3eeb: Status 404 returned error can't find the container with id 50c655b992d7171698512defea046dc582c11ec48f28fa7b5927e9592e4f3eeb
Jul 17 13:19:52.145140 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:52.145114    2569 generic.go:356] "Generic (PLEG): container finished" podID="6da58a42-233c-4f0c-a2ca-9415dee1b2f3" containerID="37a2dabbea9caf5a4fe1726358c934a97b97e81e9d9d7d1a8724d2029873c0c5" exitCode=6
Jul 17 13:19:52.145503 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:52.145186    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr" event={"ID":"6da58a42-233c-4f0c-a2ca-9415dee1b2f3","Type":"ContainerDied","Data":"37a2dabbea9caf5a4fe1726358c934a97b97e81e9d9d7d1a8724d2029873c0c5"}
Jul 17 13:19:52.145503 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:52.145232    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr" event={"ID":"6da58a42-233c-4f0c-a2ca-9415dee1b2f3","Type":"ContainerStarted","Data":"50c655b992d7171698512defea046dc582c11ec48f28fa7b5927e9592e4f3eeb"}
Jul 17 13:19:52.145503 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:52.145358    2569 scope.go:117] "RemoveContainer" containerID="37a2dabbea9caf5a4fe1726358c934a97b97e81e9d9d7d1a8724d2029873c0c5"
Jul 17 13:19:53.151466 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:53.151429    2569 generic.go:356] "Generic (PLEG): container finished" podID="6da58a42-233c-4f0c-a2ca-9415dee1b2f3" containerID="d8b83869ac4fc25ac18e84130cc8cc950e38c89545d1a33533832e922448ff8e" exitCode=6
Jul 17 13:19:53.151861 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:53.151497    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr" event={"ID":"6da58a42-233c-4f0c-a2ca-9415dee1b2f3","Type":"ContainerDied","Data":"d8b83869ac4fc25ac18e84130cc8cc950e38c89545d1a33533832e922448ff8e"}
Jul 17 13:19:53.151861 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:53.151559    2569 scope.go:117] "RemoveContainer" containerID="37a2dabbea9caf5a4fe1726358c934a97b97e81e9d9d7d1a8724d2029873c0c5"
Jul 17 13:19:53.151861 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:53.151725    2569 scope.go:117] "RemoveContainer" containerID="d8b83869ac4fc25ac18e84130cc8cc950e38c89545d1a33533832e922448ff8e"
Jul 17 13:19:53.152018 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:19:53.151971    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr_odh-ai-gateway-infra(6da58a42-233c-4f0c-a2ca-9415dee1b2f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr" podUID="6da58a42-233c-4f0c-a2ca-9415dee1b2f3"
Jul 17 13:19:54.157444 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:54.157412    2569 scope.go:117] "RemoveContainer" containerID="d8b83869ac4fc25ac18e84130cc8cc950e38c89545d1a33533832e922448ff8e"
Jul 17 13:19:54.157849 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:19:54.157575    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr_odh-ai-gateway-infra(6da58a42-233c-4f0c-a2ca-9415dee1b2f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr" podUID="6da58a42-233c-4f0c-a2ca-9415dee1b2f3"
Jul 17 13:19:56.628488 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:56.628459    2569 scope.go:117] "RemoveContainer" containerID="cc2df862e47f35b1130d7da0dc707b33bbd8390601fcc7d8ef3d18cb5d07d8de"
Jul 17 13:19:57.173032 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:57.173006    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t" event={"ID":"05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8","Type":"ContainerStarted","Data":"4b69554e21285c822b1ff90f6d730d57d9c5b95c59f851eaddb41d75d40cb7a6"}
Jul 17 13:19:57.194049 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:57.194006    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t" podStartSLOduration=18.193991691 podStartE2EDuration="18.193991691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:19:57.191636431 +0000 UTC m=+3472.195683313" watchObservedRunningTime="2026-07-17 13:19:57.193991691 +0000 UTC m=+3472.198038624"
Jul 17 13:19:58.180131 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:58.180093    2569 generic.go:356] "Generic (PLEG): container finished" podID="05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8" containerID="4b69554e21285c822b1ff90f6d730d57d9c5b95c59f851eaddb41d75d40cb7a6" exitCode=6
Jul 17 13:19:58.180536 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:58.180174    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t" event={"ID":"05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8","Type":"ContainerDied","Data":"4b69554e21285c822b1ff90f6d730d57d9c5b95c59f851eaddb41d75d40cb7a6"}
Jul 17 13:19:58.180536 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:58.180244    2569 scope.go:117] "RemoveContainer" containerID="cc2df862e47f35b1130d7da0dc707b33bbd8390601fcc7d8ef3d18cb5d07d8de"
Jul 17 13:19:58.180536 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:58.180488    2569 scope.go:117] "RemoveContainer" containerID="4b69554e21285c822b1ff90f6d730d57d9c5b95c59f851eaddb41d75d40cb7a6"
Jul 17 13:19:58.180716 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:19:58.180695    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t_odh-ai-gateway-infra(05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t" podUID="05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8"
Jul 17 13:19:58.223825 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:58.223799    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t"]
Jul 17 13:19:59.312432 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:59.312405    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t"
Jul 17 13:19:59.397813 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:59.397785    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8-maas-api-service-ca\") pod \"05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8\" (UID: \"05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8\") "
Jul 17 13:19:59.398101 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:59.398077    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8" (UID: "05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:19:59.499126 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:19:59.499103    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:20:00.191931 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:00.191898    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t" event={"ID":"05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8","Type":"ContainerDied","Data":"a74396677da74d439c961629629f25d20ed80275b6bfef1a0b35370ff9d08fd1"}
Jul 17 13:20:00.191931 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:00.191924    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t"
Jul 17 13:20:00.192126 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:00.191960    2569 scope.go:117] "RemoveContainer" containerID="4b69554e21285c822b1ff90f6d730d57d9c5b95c59f851eaddb41d75d40cb7a6"
Jul 17 13:20:00.210143 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:00.210110    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t"]
Jul 17 13:20:00.212116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:00.212095    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bjt6t"]
Jul 17 13:20:01.633491 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:01.633459    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8" path="/var/lib/kubelet/pods/05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8/volumes"
Jul 17 13:20:02.251744 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:02.251715    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9"]
Jul 17 13:20:02.252171 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:02.252151    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8" containerName="revoke-keys"
Jul 17 13:20:02.252224 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:02.252174    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8" containerName="revoke-keys"
Jul 17 13:20:02.252224 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:02.252207    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8" containerName="revoke-keys"
Jul 17 13:20:02.252224 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:02.252217    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8" containerName="revoke-keys"
Jul 17 13:20:02.252348 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:02.252291    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8" containerName="revoke-keys"
Jul 17 13:20:02.252348 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:02.252302    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8" containerName="revoke-keys"
Jul 17 13:20:02.252348 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:02.252312    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8" containerName="revoke-keys"
Jul 17 13:20:02.256936 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:02.256921    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9"
Jul 17 13:20:02.266883 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:02.266859    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9"]
Jul 17 13:20:02.321987 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:02.321959    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e0afb9-cfec-4361-9741-c990bb73ada5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9\" (UID: \"40e0afb9-cfec-4361-9741-c990bb73ada5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9"
Jul 17 13:20:02.422568 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:02.422525    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e0afb9-cfec-4361-9741-c990bb73ada5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9\" (UID: \"40e0afb9-cfec-4361-9741-c990bb73ada5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9"
Jul 17 13:20:02.423162 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:02.423144    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e0afb9-cfec-4361-9741-c990bb73ada5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9\" (UID: \"40e0afb9-cfec-4361-9741-c990bb73ada5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9"
Jul 17 13:20:02.567618 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:02.567544    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9"
Jul 17 13:20:02.695198 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:02.695174    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9"]
Jul 17 13:20:02.697320 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:20:02.697289    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e0afb9_cfec_4361_9741_c990bb73ada5.slice/crio-aa0670e7752be118bddd862cfedc94fb1b3482a9e1d90fd581169f72546de525 WatchSource:0}: Error finding container aa0670e7752be118bddd862cfedc94fb1b3482a9e1d90fd581169f72546de525: Status 404 returned error can't find the container with id aa0670e7752be118bddd862cfedc94fb1b3482a9e1d90fd581169f72546de525
Jul 17 13:20:03.208299 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:03.208216    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9" event={"ID":"40e0afb9-cfec-4361-9741-c990bb73ada5","Type":"ContainerStarted","Data":"66893629f40c8907aeb71e5b1b60c09fa88b56d23370adab2ffbcdfc1202f90e"}
Jul 17 13:20:03.208299 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:03.208289    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9" event={"ID":"40e0afb9-cfec-4361-9741-c990bb73ada5","Type":"ContainerStarted","Data":"aa0670e7752be118bddd862cfedc94fb1b3482a9e1d90fd581169f72546de525"}
Jul 17 13:20:03.226081 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:03.226033    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9" podStartSLOduration=1.226019223 podStartE2EDuration="1.226019223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:20:03.225357992 +0000 UTC m=+3478.229404874" watchObservedRunningTime="2026-07-17 13:20:03.226019223 +0000 UTC m=+3478.230066104"
Jul 17 13:20:04.214520 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:04.214486    2569 generic.go:356] "Generic (PLEG): container finished" podID="40e0afb9-cfec-4361-9741-c990bb73ada5" containerID="66893629f40c8907aeb71e5b1b60c09fa88b56d23370adab2ffbcdfc1202f90e" exitCode=6
Jul 17 13:20:04.214938 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:04.214567    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9" event={"ID":"40e0afb9-cfec-4361-9741-c990bb73ada5","Type":"ContainerDied","Data":"66893629f40c8907aeb71e5b1b60c09fa88b56d23370adab2ffbcdfc1202f90e"}
Jul 17 13:20:04.214938 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:04.214726    2569 scope.go:117] "RemoveContainer" containerID="66893629f40c8907aeb71e5b1b60c09fa88b56d23370adab2ffbcdfc1202f90e"
Jul 17 13:20:05.221104 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:05.221019    2569 generic.go:356] "Generic (PLEG): container finished" podID="40e0afb9-cfec-4361-9741-c990bb73ada5" containerID="ce9966d06ed91087933c1c2cb2f6039384012ba66b4b249397c0973348bf921c" exitCode=6
Jul 17 13:20:05.221497 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:05.221111    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9" event={"ID":"40e0afb9-cfec-4361-9741-c990bb73ada5","Type":"ContainerDied","Data":"ce9966d06ed91087933c1c2cb2f6039384012ba66b4b249397c0973348bf921c"}
Jul 17 13:20:05.221497 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:05.221166    2569 scope.go:117] "RemoveContainer" containerID="66893629f40c8907aeb71e5b1b60c09fa88b56d23370adab2ffbcdfc1202f90e"
Jul 17 13:20:05.221497 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:05.221275    2569 scope.go:117] "RemoveContainer" containerID="ce9966d06ed91087933c1c2cb2f6039384012ba66b4b249397c0973348bf921c"
Jul 17 13:20:05.221622 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:20:05.221493    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9_odh-ai-gateway-infra(40e0afb9-cfec-4361-9741-c990bb73ada5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9" podUID="40e0afb9-cfec-4361-9741-c990bb73ada5"
Jul 17 13:20:06.227161 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:06.227141    2569 scope.go:117] "RemoveContainer" containerID="ce9966d06ed91087933c1c2cb2f6039384012ba66b4b249397c0973348bf921c"
Jul 17 13:20:06.227496 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:20:06.227313    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9_odh-ai-gateway-infra(40e0afb9-cfec-4361-9741-c990bb73ada5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9" podUID="40e0afb9-cfec-4361-9741-c990bb73ada5"
Jul 17 13:20:07.628372 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:07.628345    2569 scope.go:117] "RemoveContainer" containerID="d8b83869ac4fc25ac18e84130cc8cc950e38c89545d1a33533832e922448ff8e"
Jul 17 13:20:08.237283 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:08.237256    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr" event={"ID":"6da58a42-233c-4f0c-a2ca-9415dee1b2f3","Type":"ContainerStarted","Data":"7bde3c9691b2c81a6b11bc8e24b145ba295d2c413ab21703b85cf2145fcafeb9"}
Jul 17 13:20:08.263216 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:08.263170    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr" podStartSLOduration=17.26315505 podStartE2EDuration="17.26315505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:20:08.261439783 +0000 UTC m=+3483.265486664" watchObservedRunningTime="2026-07-17 13:20:08.26315505 +0000 UTC m=+3483.267201933"
Jul 17 13:20:09.244484 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:09.244439    2569 generic.go:356] "Generic (PLEG): container finished" podID="6da58a42-233c-4f0c-a2ca-9415dee1b2f3" containerID="7bde3c9691b2c81a6b11bc8e24b145ba295d2c413ab21703b85cf2145fcafeb9" exitCode=6
Jul 17 13:20:09.244484 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:09.244479    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr" event={"ID":"6da58a42-233c-4f0c-a2ca-9415dee1b2f3","Type":"ContainerDied","Data":"7bde3c9691b2c81a6b11bc8e24b145ba295d2c413ab21703b85cf2145fcafeb9"}
Jul 17 13:20:09.245044 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:09.244541    2569 scope.go:117] "RemoveContainer" containerID="d8b83869ac4fc25ac18e84130cc8cc950e38c89545d1a33533832e922448ff8e"
Jul 17 13:20:09.245044 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:09.244781    2569 scope.go:117] "RemoveContainer" containerID="7bde3c9691b2c81a6b11bc8e24b145ba295d2c413ab21703b85cf2145fcafeb9"
Jul 17 13:20:09.245044 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:20:09.245031    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr_odh-ai-gateway-infra(6da58a42-233c-4f0c-a2ca-9415dee1b2f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr" podUID="6da58a42-233c-4f0c-a2ca-9415dee1b2f3"
Jul 17 13:20:09.276800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:09.276768    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr"]
Jul 17 13:20:10.388312 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:10.388289    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr"
Jul 17 13:20:10.485376 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:10.485350    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da58a42-233c-4f0c-a2ca-9415dee1b2f3-maas-api-service-ca\") pod \"6da58a42-233c-4f0c-a2ca-9415dee1b2f3\" (UID: \"6da58a42-233c-4f0c-a2ca-9415dee1b2f3\") "
Jul 17 13:20:10.485709 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:10.485687    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da58a42-233c-4f0c-a2ca-9415dee1b2f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6da58a42-233c-4f0c-a2ca-9415dee1b2f3" (UID: "6da58a42-233c-4f0c-a2ca-9415dee1b2f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:20:10.586226 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:10.586169    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da58a42-233c-4f0c-a2ca-9415dee1b2f3-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:20:11.258244 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:11.258217    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr"
Jul 17 13:20:11.258371 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:11.258244    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr" event={"ID":"6da58a42-233c-4f0c-a2ca-9415dee1b2f3","Type":"ContainerDied","Data":"50c655b992d7171698512defea046dc582c11ec48f28fa7b5927e9592e4f3eeb"}
Jul 17 13:20:11.258371 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:11.258310    2569 scope.go:117] "RemoveContainer" containerID="7bde3c9691b2c81a6b11bc8e24b145ba295d2c413ab21703b85cf2145fcafeb9"
Jul 17 13:20:11.282894 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:11.282859    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr"]
Jul 17 13:20:11.288732 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:11.288702    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-cdrgr"]
Jul 17 13:20:11.634277 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:11.634214    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6da58a42-233c-4f0c-a2ca-9415dee1b2f3" path="/var/lib/kubelet/pods/6da58a42-233c-4f0c-a2ca-9415dee1b2f3/volumes"
Jul 17 13:20:17.168636 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:17.168596    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:20:17.633935 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:17.633910    2569 scope.go:117] "RemoveContainer" containerID="ce9966d06ed91087933c1c2cb2f6039384012ba66b4b249397c0973348bf921c"
Jul 17 13:20:18.295475 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:18.295440    2569 generic.go:356] "Generic (PLEG): container finished" podID="40e0afb9-cfec-4361-9741-c990bb73ada5" containerID="4e0e7b8e84e8f7f4e4fb86e7945ec5ea70982549a66a0ece7bee9af7fe5cacb4" exitCode=6
Jul 17 13:20:18.295802 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:18.295513    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9" event={"ID":"40e0afb9-cfec-4361-9741-c990bb73ada5","Type":"ContainerDied","Data":"4e0e7b8e84e8f7f4e4fb86e7945ec5ea70982549a66a0ece7bee9af7fe5cacb4"}
Jul 17 13:20:18.295802 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:18.295575    2569 scope.go:117] "RemoveContainer" containerID="ce9966d06ed91087933c1c2cb2f6039384012ba66b4b249397c0973348bf921c"
Jul 17 13:20:18.295940 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:18.295923    2569 scope.go:117] "RemoveContainer" containerID="4e0e7b8e84e8f7f4e4fb86e7945ec5ea70982549a66a0ece7bee9af7fe5cacb4"
Jul 17 13:20:18.296118 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:20:18.296097    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9_odh-ai-gateway-infra(40e0afb9-cfec-4361-9741-c990bb73ada5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9" podUID="40e0afb9-cfec-4361-9741-c990bb73ada5"
Jul 17 13:20:19.322359 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:19.322326    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9"]
Jul 17 13:20:19.456878 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:19.456856    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9"
Jul 17 13:20:19.553515 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:19.553486    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e0afb9-cfec-4361-9741-c990bb73ada5-maas-api-service-ca\") pod \"40e0afb9-cfec-4361-9741-c990bb73ada5\" (UID: \"40e0afb9-cfec-4361-9741-c990bb73ada5\") "
Jul 17 13:20:19.553844 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:19.553825    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e0afb9-cfec-4361-9741-c990bb73ada5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "40e0afb9-cfec-4361-9741-c990bb73ada5" (UID: "40e0afb9-cfec-4361-9741-c990bb73ada5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:20:19.654456 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:19.654430    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e0afb9-cfec-4361-9741-c990bb73ada5-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:20:20.309615 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:20.309582    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9"
Jul 17 13:20:20.309801 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:20.309608    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9" event={"ID":"40e0afb9-cfec-4361-9741-c990bb73ada5","Type":"ContainerDied","Data":"aa0670e7752be118bddd862cfedc94fb1b3482a9e1d90fd581169f72546de525"}
Jul 17 13:20:20.309801 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:20.309683    2569 scope.go:117] "RemoveContainer" containerID="4e0e7b8e84e8f7f4e4fb86e7945ec5ea70982549a66a0ece7bee9af7fe5cacb4"
Jul 17 13:20:20.349328 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:20.349301    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9"]
Jul 17 13:20:20.366881 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:20.366853    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-dc9t9"]
Jul 17 13:20:21.634202 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:21.634168    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40e0afb9-cfec-4361-9741-c990bb73ada5" path="/var/lib/kubelet/pods/40e0afb9-cfec-4361-9741-c990bb73ada5/volumes"
Jul 17 13:20:36.623770 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:36.623724    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:20:38.652086 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.652051    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x"]
Jul 17 13:20:38.652525 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.652357    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8" containerName="revoke-keys"
Jul 17 13:20:38.652525 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.652368    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="05bb3cca-b8a4-4a3d-bb4a-9b51dd13eec8" containerName="revoke-keys"
Jul 17 13:20:38.652525 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.652379    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6da58a42-233c-4f0c-a2ca-9415dee1b2f3" containerName="revoke-keys"
Jul 17 13:20:38.652525 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.652385    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6da58a42-233c-4f0c-a2ca-9415dee1b2f3" containerName="revoke-keys"
Jul 17 13:20:38.652525 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.652391    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e0afb9-cfec-4361-9741-c990bb73ada5" containerName="revoke-keys"
Jul 17 13:20:38.652525 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.652397    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e0afb9-cfec-4361-9741-c990bb73ada5" containerName="revoke-keys"
Jul 17 13:20:38.652525 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.652405    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e0afb9-cfec-4361-9741-c990bb73ada5" containerName="revoke-keys"
Jul 17 13:20:38.652525 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.652409    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e0afb9-cfec-4361-9741-c990bb73ada5" containerName="revoke-keys"
Jul 17 13:20:38.652525 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.652421    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6da58a42-233c-4f0c-a2ca-9415dee1b2f3" containerName="revoke-keys"
Jul 17 13:20:38.652525 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.652425    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6da58a42-233c-4f0c-a2ca-9415dee1b2f3" containerName="revoke-keys"
Jul 17 13:20:38.652525 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.652482    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6da58a42-233c-4f0c-a2ca-9415dee1b2f3" containerName="revoke-keys"
Jul 17 13:20:38.652525 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.652488    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="40e0afb9-cfec-4361-9741-c990bb73ada5" containerName="revoke-keys"
Jul 17 13:20:38.652525 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.652494    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6da58a42-233c-4f0c-a2ca-9415dee1b2f3" containerName="revoke-keys"
Jul 17 13:20:38.652525 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.652501    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="40e0afb9-cfec-4361-9741-c990bb73ada5" containerName="revoke-keys"
Jul 17 13:20:38.652525 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.652508    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="40e0afb9-cfec-4361-9741-c990bb73ada5" containerName="revoke-keys"
Jul 17 13:20:38.656553 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.656533    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x"
Jul 17 13:20:38.659836 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.659808    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:20:38.666481 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.666458    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x"]
Jul 17 13:20:38.793906 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.793880    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7880e8-010b-4e9f-98b1-c4be084947c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x\" (UID: \"9e7880e8-010b-4e9f-98b1-c4be084947c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x"
Jul 17 13:20:38.894408 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.894379    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7880e8-010b-4e9f-98b1-c4be084947c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x\" (UID: \"9e7880e8-010b-4e9f-98b1-c4be084947c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x"
Jul 17 13:20:38.894975 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.894953    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7880e8-010b-4e9f-98b1-c4be084947c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x\" (UID: \"9e7880e8-010b-4e9f-98b1-c4be084947c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x"
Jul 17 13:20:38.967501 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:38.967448    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x"
Jul 17 13:20:39.100309 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:39.100287    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x"]
Jul 17 13:20:39.102231 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:20:39.102200    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7880e8_010b_4e9f_98b1_c4be084947c7.slice/crio-88b43a1addad695be40dfc3a90ba3dbcde49fc10ffa5565bfaa06eeec2b0e1f9 WatchSource:0}: Error finding container 88b43a1addad695be40dfc3a90ba3dbcde49fc10ffa5565bfaa06eeec2b0e1f9: Status 404 returned error can't find the container with id 88b43a1addad695be40dfc3a90ba3dbcde49fc10ffa5565bfaa06eeec2b0e1f9
Jul 17 13:20:39.403430 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:39.403399    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x" event={"ID":"9e7880e8-010b-4e9f-98b1-c4be084947c7","Type":"ContainerStarted","Data":"b9eea22d5dc3bd8356375c546a77abcbbd2ef0dc341094d46ba3bccbcc8a3945"}
Jul 17 13:20:39.403526 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:39.403457    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x" event={"ID":"9e7880e8-010b-4e9f-98b1-c4be084947c7","Type":"ContainerStarted","Data":"88b43a1addad695be40dfc3a90ba3dbcde49fc10ffa5565bfaa06eeec2b0e1f9"}
Jul 17 13:20:39.421999 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:39.421954    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x" podStartSLOduration=1.421939487 podStartE2EDuration="1.421939487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:20:39.420681457 +0000 UTC m=+3514.424728341" watchObservedRunningTime="2026-07-17 13:20:39.421939487 +0000 UTC m=+3514.425986369"
Jul 17 13:20:39.916526 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:39.916496    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh"]
Jul 17 13:20:39.916875 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:39.916859    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6da58a42-233c-4f0c-a2ca-9415dee1b2f3" containerName="revoke-keys"
Jul 17 13:20:39.916875 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:39.916875    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6da58a42-233c-4f0c-a2ca-9415dee1b2f3" containerName="revoke-keys"
Jul 17 13:20:39.916960 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:39.916888    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e0afb9-cfec-4361-9741-c990bb73ada5" containerName="revoke-keys"
Jul 17 13:20:39.916960 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:39.916894    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e0afb9-cfec-4361-9741-c990bb73ada5" containerName="revoke-keys"
Jul 17 13:20:39.916960 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:39.916957    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6da58a42-233c-4f0c-a2ca-9415dee1b2f3" containerName="revoke-keys"
Jul 17 13:20:39.919934 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:39.919917    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh"
Jul 17 13:20:39.927701 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:39.927674    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh"]
Jul 17 13:20:40.003606 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:40.003555    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0a31c-6a77-466f-b083-397ead8621b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh\" (UID: \"e9d0a31c-6a77-466f-b083-397ead8621b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh"
Jul 17 13:20:40.104668 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:40.104644    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0a31c-6a77-466f-b083-397ead8621b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh\" (UID: \"e9d0a31c-6a77-466f-b083-397ead8621b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh"
Jul 17 13:20:40.105199 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:40.105182    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0a31c-6a77-466f-b083-397ead8621b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh\" (UID: \"e9d0a31c-6a77-466f-b083-397ead8621b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh"
Jul 17 13:20:40.231387 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:40.231355    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh"
Jul 17 13:20:40.363289 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:40.363257    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh"]
Jul 17 13:20:40.366205 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:20:40.366176    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d0a31c_6a77_466f_b083_397ead8621b8.slice/crio-cc2138e5eea23163153200d3a979b42cfecfb53ec11f1ec6f2a3151cf6edc389 WatchSource:0}: Error finding container cc2138e5eea23163153200d3a979b42cfecfb53ec11f1ec6f2a3151cf6edc389: Status 404 returned error can't find the container with id cc2138e5eea23163153200d3a979b42cfecfb53ec11f1ec6f2a3151cf6edc389
Jul 17 13:20:40.408677 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:40.408657    2569 generic.go:356] "Generic (PLEG): container finished" podID="9e7880e8-010b-4e9f-98b1-c4be084947c7" containerID="b9eea22d5dc3bd8356375c546a77abcbbd2ef0dc341094d46ba3bccbcc8a3945" exitCode=6
Jul 17 13:20:40.408801 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:40.408734    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x" event={"ID":"9e7880e8-010b-4e9f-98b1-c4be084947c7","Type":"ContainerDied","Data":"b9eea22d5dc3bd8356375c546a77abcbbd2ef0dc341094d46ba3bccbcc8a3945"}
Jul 17 13:20:40.408943 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:40.408927    2569 scope.go:117] "RemoveContainer" containerID="b9eea22d5dc3bd8356375c546a77abcbbd2ef0dc341094d46ba3bccbcc8a3945"
Jul 17 13:20:40.410008 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:40.409990    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh" event={"ID":"e9d0a31c-6a77-466f-b083-397ead8621b8","Type":"ContainerStarted","Data":"cc2138e5eea23163153200d3a979b42cfecfb53ec11f1ec6f2a3151cf6edc389"}
Jul 17 13:20:41.416557 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:41.416491    2569 generic.go:356] "Generic (PLEG): container finished" podID="e9d0a31c-6a77-466f-b083-397ead8621b8" containerID="f3a37034af267d5758b87c3ef0282c2cd33ef3e5460bff8527a5f4dd42ec4fae" exitCode=6
Jul 17 13:20:41.416959 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:41.416577    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh" event={"ID":"e9d0a31c-6a77-466f-b083-397ead8621b8","Type":"ContainerDied","Data":"f3a37034af267d5758b87c3ef0282c2cd33ef3e5460bff8527a5f4dd42ec4fae"}
Jul 17 13:20:41.416959 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:41.416783    2569 scope.go:117] "RemoveContainer" containerID="f3a37034af267d5758b87c3ef0282c2cd33ef3e5460bff8527a5f4dd42ec4fae"
Jul 17 13:20:41.418881 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:41.418772    2569 generic.go:356] "Generic (PLEG): container finished" podID="9e7880e8-010b-4e9f-98b1-c4be084947c7" containerID="63c88ec474f3effa3e5f8dee173eb05dd76704674de47ebb3288f5bbd599da86" exitCode=6
Jul 17 13:20:41.418881 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:41.418816    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x" event={"ID":"9e7880e8-010b-4e9f-98b1-c4be084947c7","Type":"ContainerDied","Data":"63c88ec474f3effa3e5f8dee173eb05dd76704674de47ebb3288f5bbd599da86"}
Jul 17 13:20:41.418881 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:41.418866    2569 scope.go:117] "RemoveContainer" containerID="b9eea22d5dc3bd8356375c546a77abcbbd2ef0dc341094d46ba3bccbcc8a3945"
Jul 17 13:20:41.419028 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:41.419008    2569 scope.go:117] "RemoveContainer" containerID="63c88ec474f3effa3e5f8dee173eb05dd76704674de47ebb3288f5bbd599da86"
Jul 17 13:20:41.419227 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:20:41.419200    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x_odh-ai-gateway-infra(9e7880e8-010b-4e9f-98b1-c4be084947c7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x" podUID="9e7880e8-010b-4e9f-98b1-c4be084947c7"
Jul 17 13:20:42.425445 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:42.425380    2569 generic.go:356] "Generic (PLEG): container finished" podID="e9d0a31c-6a77-466f-b083-397ead8621b8" containerID="7af5ee24c6f63837c2e0bb4c17523cbdaddd6a85039b19f052b4d2d71502a217" exitCode=6
Jul 17 13:20:42.425821 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:42.425470    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh" event={"ID":"e9d0a31c-6a77-466f-b083-397ead8621b8","Type":"ContainerDied","Data":"7af5ee24c6f63837c2e0bb4c17523cbdaddd6a85039b19f052b4d2d71502a217"}
Jul 17 13:20:42.425821 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:42.425555    2569 scope.go:117] "RemoveContainer" containerID="f3a37034af267d5758b87c3ef0282c2cd33ef3e5460bff8527a5f4dd42ec4fae"
Jul 17 13:20:42.425821 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:42.425642    2569 scope.go:117] "RemoveContainer" containerID="7af5ee24c6f63837c2e0bb4c17523cbdaddd6a85039b19f052b4d2d71502a217"
Jul 17 13:20:42.425985 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:20:42.425852    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh_odh-ai-gateway-infra(e9d0a31c-6a77-466f-b083-397ead8621b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh" podUID="e9d0a31c-6a77-466f-b083-397ead8621b8"
Jul 17 13:20:42.427552 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:42.427536    2569 scope.go:117] "RemoveContainer" containerID="63c88ec474f3effa3e5f8dee173eb05dd76704674de47ebb3288f5bbd599da86"
Jul 17 13:20:42.427713 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:20:42.427685    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x_odh-ai-gateway-infra(9e7880e8-010b-4e9f-98b1-c4be084947c7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x" podUID="9e7880e8-010b-4e9f-98b1-c4be084947c7"
Jul 17 13:20:43.434241 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:43.434212    2569 scope.go:117] "RemoveContainer" containerID="7af5ee24c6f63837c2e0bb4c17523cbdaddd6a85039b19f052b4d2d71502a217"
Jul 17 13:20:43.434625 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:20:43.434386    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh_odh-ai-gateway-infra(e9d0a31c-6a77-466f-b083-397ead8621b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh" podUID="e9d0a31c-6a77-466f-b083-397ead8621b8"
Jul 17 13:20:52.299608 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:52.299572    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq"]
Jul 17 13:20:52.303125 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:52.303105    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq"
Jul 17 13:20:52.310840 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:52.310807    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq"]
Jul 17 13:20:52.384712 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:52.384679    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c948e35-18b6-454f-9167-c743fbf431d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq\" (UID: \"9c948e35-18b6-454f-9167-c743fbf431d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq"
Jul 17 13:20:52.485983 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:52.485957    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c948e35-18b6-454f-9167-c743fbf431d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq\" (UID: \"9c948e35-18b6-454f-9167-c743fbf431d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq"
Jul 17 13:20:52.486621 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:52.486599    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c948e35-18b6-454f-9167-c743fbf431d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq\" (UID: \"9c948e35-18b6-454f-9167-c743fbf431d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq"
Jul 17 13:20:52.616048 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:52.615989    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq"
Jul 17 13:20:52.768506 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:52.768100    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq"]
Jul 17 13:20:53.482808 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:53.482777    2569 generic.go:356] "Generic (PLEG): container finished" podID="9c948e35-18b6-454f-9167-c743fbf431d3" containerID="eede9be020a93e1db2e39f001a9cbe58c6fd7f11893871730b278c2bca5b1af1" exitCode=6
Jul 17 13:20:53.483132 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:53.482885    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq" event={"ID":"9c948e35-18b6-454f-9167-c743fbf431d3","Type":"ContainerDied","Data":"eede9be020a93e1db2e39f001a9cbe58c6fd7f11893871730b278c2bca5b1af1"}
Jul 17 13:20:53.483132 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:53.482928    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq" event={"ID":"9c948e35-18b6-454f-9167-c743fbf431d3","Type":"ContainerStarted","Data":"1a6f557a8a89c5bd5848a465f3c5142cc9029f480dccb7c09856f5cbc2f871a1"}
Jul 17 13:20:53.483132 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:53.483099    2569 scope.go:117] "RemoveContainer" containerID="eede9be020a93e1db2e39f001a9cbe58c6fd7f11893871730b278c2bca5b1af1"
Jul 17 13:20:54.489483 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:54.489455    2569 generic.go:356] "Generic (PLEG): container finished" podID="9c948e35-18b6-454f-9167-c743fbf431d3" containerID="ee9ead00dc05a65f122a195823a047e044d42f420ff503c2405b2a59d3d22516" exitCode=6
Jul 17 13:20:54.489985 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:54.489554    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq" event={"ID":"9c948e35-18b6-454f-9167-c743fbf431d3","Type":"ContainerDied","Data":"ee9ead00dc05a65f122a195823a047e044d42f420ff503c2405b2a59d3d22516"}
Jul 17 13:20:54.489985 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:54.489627    2569 scope.go:117] "RemoveContainer" containerID="eede9be020a93e1db2e39f001a9cbe58c6fd7f11893871730b278c2bca5b1af1"
Jul 17 13:20:54.489985 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:54.489693    2569 scope.go:117] "RemoveContainer" containerID="ee9ead00dc05a65f122a195823a047e044d42f420ff503c2405b2a59d3d22516"
Jul 17 13:20:54.489985 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:20:54.489873    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq_odh-ai-gateway-infra(9c948e35-18b6-454f-9167-c743fbf431d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq" podUID="9c948e35-18b6-454f-9167-c743fbf431d3"
Jul 17 13:20:54.628154 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:54.628134    2569 scope.go:117] "RemoveContainer" containerID="63c88ec474f3effa3e5f8dee173eb05dd76704674de47ebb3288f5bbd599da86"
Jul 17 13:20:55.397533 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:55.397500    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"]
Jul 17 13:20:55.397818 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:55.397786    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9" podUID="e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd" containerName="maas-api" containerID="cri-o://1f276333ba7dbe86b40167177b8bddc6e5b6a4d8c9ad910152b8db447bde3842" gracePeriod=30
Jul 17 13:20:55.495569 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:55.495545    2569 generic.go:356] "Generic (PLEG): container finished" podID="9e7880e8-010b-4e9f-98b1-c4be084947c7" containerID="6620804f839b3ffd942f5457d8c10062657aa2f3c30cdacec4fdf22659894e38" exitCode=6
Jul 17 13:20:55.496141 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:55.495616    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x" event={"ID":"9e7880e8-010b-4e9f-98b1-c4be084947c7","Type":"ContainerDied","Data":"6620804f839b3ffd942f5457d8c10062657aa2f3c30cdacec4fdf22659894e38"}
Jul 17 13:20:55.496141 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:55.495671    2569 scope.go:117] "RemoveContainer" containerID="63c88ec474f3effa3e5f8dee173eb05dd76704674de47ebb3288f5bbd599da86"
Jul 17 13:20:55.496141 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:55.495938    2569 scope.go:117] "RemoveContainer" containerID="6620804f839b3ffd942f5457d8c10062657aa2f3c30cdacec4fdf22659894e38"
Jul 17 13:20:55.496336 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:20:55.496161    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x_odh-ai-gateway-infra(9e7880e8-010b-4e9f-98b1-c4be084947c7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x" podUID="9e7880e8-010b-4e9f-98b1-c4be084947c7"
Jul 17 13:20:55.497461 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:55.497444    2569 scope.go:117] "RemoveContainer" containerID="ee9ead00dc05a65f122a195823a047e044d42f420ff503c2405b2a59d3d22516"
Jul 17 13:20:55.497601 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:20:55.497585    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq_odh-ai-gateway-infra(9c948e35-18b6-454f-9167-c743fbf431d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq" podUID="9c948e35-18b6-454f-9167-c743fbf431d3"
Jul 17 13:20:55.646936 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:55.646908    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"
Jul 17 13:20:55.711275 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:55.711250    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd-maas-api-tls\") pod \"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd\" (UID: \"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd\") "
Jul 17 13:20:55.711405 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:55.711285    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fm59b\" (UniqueName: \"kubernetes.io/projected/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd-kube-api-access-fm59b\") pod \"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd\" (UID: \"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd\") "
Jul 17 13:20:55.711405 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:55.711316    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd-openshift-service-ca\") pod \"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd\" (UID: \"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd\") "
Jul 17 13:20:55.711739 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:55.711704    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd" (UID: "e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:20:55.713877 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:55.713851    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd" (UID: "e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:20:55.713877 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:55.713871    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd-kube-api-access-fm59b" (OuterVolumeSpecName: "kube-api-access-fm59b") pod "e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd" (UID: "e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd"). InnerVolumeSpecName "kube-api-access-fm59b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:20:55.812167 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:55.812108    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm59b\" (UniqueName: \"kubernetes.io/projected/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd-kube-api-access-fm59b\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:20:55.812167 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:55.812130    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd-openshift-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:20:55.812167 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:55.812139    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd-maas-api-tls\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:20:56.506715 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:56.506681    2569 generic.go:356] "Generic (PLEG): container finished" podID="e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd" containerID="1f276333ba7dbe86b40167177b8bddc6e5b6a4d8c9ad910152b8db447bde3842" exitCode=0
Jul 17 13:20:56.507113 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:56.506713    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9" event={"ID":"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd","Type":"ContainerDied","Data":"1f276333ba7dbe86b40167177b8bddc6e5b6a4d8c9ad910152b8db447bde3842"}
Jul 17 13:20:56.507113 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:56.506741    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"
Jul 17 13:20:56.507113 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:56.506786    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9" event={"ID":"e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd","Type":"ContainerDied","Data":"a717950ee557627fc387ad0465bea190b4c288e6dfbf7059ed9f00bc9b58cb51"}
Jul 17 13:20:56.507113 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:56.506812    2569 scope.go:117] "RemoveContainer" containerID="1f276333ba7dbe86b40167177b8bddc6e5b6a4d8c9ad910152b8db447bde3842"
Jul 17 13:20:56.515297 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:56.515279    2569 scope.go:117] "RemoveContainer" containerID="1f276333ba7dbe86b40167177b8bddc6e5b6a4d8c9ad910152b8db447bde3842"
Jul 17 13:20:56.515543 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:20:56.515525    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f276333ba7dbe86b40167177b8bddc6e5b6a4d8c9ad910152b8db447bde3842\": container with ID starting with 1f276333ba7dbe86b40167177b8bddc6e5b6a4d8c9ad910152b8db447bde3842 not found: ID does not exist" containerID="1f276333ba7dbe86b40167177b8bddc6e5b6a4d8c9ad910152b8db447bde3842"
Jul 17 13:20:56.515592 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:56.515551    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f276333ba7dbe86b40167177b8bddc6e5b6a4d8c9ad910152b8db447bde3842"} err="failed to get container status \"1f276333ba7dbe86b40167177b8bddc6e5b6a4d8c9ad910152b8db447bde3842\": rpc error: code = NotFound desc = could not find container \"1f276333ba7dbe86b40167177b8bddc6e5b6a4d8c9ad910152b8db447bde3842\": container with ID starting with 1f276333ba7dbe86b40167177b8bddc6e5b6a4d8c9ad910152b8db447bde3842 not found: ID does not exist"
Jul 17 13:20:56.531611 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:56.531585    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"]
Jul 17 13:20:56.537156 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:56.537138    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6ad1d9a1-694c5c668d-26gd9"]
Jul 17 13:20:56.539618 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:56.539596    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x"]
Jul 17 13:20:56.628481 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:56.628455    2569 scope.go:117] "RemoveContainer" containerID="7af5ee24c6f63837c2e0bb4c17523cbdaddd6a85039b19f052b4d2d71502a217"
Jul 17 13:20:56.681220 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:56.681200    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x"
Jul 17 13:20:56.719981 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:56.719958    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7880e8-010b-4e9f-98b1-c4be084947c7-maas-api-service-ca\") pod \"9e7880e8-010b-4e9f-98b1-c4be084947c7\" (UID: \"9e7880e8-010b-4e9f-98b1-c4be084947c7\") "
Jul 17 13:20:56.720384 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:56.720357    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7880e8-010b-4e9f-98b1-c4be084947c7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9e7880e8-010b-4e9f-98b1-c4be084947c7" (UID: "9e7880e8-010b-4e9f-98b1-c4be084947c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:20:56.820543 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:56.820486    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7880e8-010b-4e9f-98b1-c4be084947c7-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:20:57.512677 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:57.512620    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x"
Jul 17 13:20:57.512677 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:57.512647    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x" event={"ID":"9e7880e8-010b-4e9f-98b1-c4be084947c7","Type":"ContainerDied","Data":"88b43a1addad695be40dfc3a90ba3dbcde49fc10ffa5565bfaa06eeec2b0e1f9"}
Jul 17 13:20:57.513148 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:57.512713    2569 scope.go:117] "RemoveContainer" containerID="6620804f839b3ffd942f5457d8c10062657aa2f3c30cdacec4fdf22659894e38"
Jul 17 13:20:57.515887 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:57.515860    2569 generic.go:356] "Generic (PLEG): container finished" podID="e9d0a31c-6a77-466f-b083-397ead8621b8" containerID="1cd181aa1c7877618b10feebf263eb77128eb8ddc06045a713ee6fb54ee067a4" exitCode=6
Jul 17 13:20:57.516000 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:57.515915    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh" event={"ID":"e9d0a31c-6a77-466f-b083-397ead8621b8","Type":"ContainerDied","Data":"1cd181aa1c7877618b10feebf263eb77128eb8ddc06045a713ee6fb54ee067a4"}
Jul 17 13:20:57.516245 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:57.516228    2569 scope.go:117] "RemoveContainer" containerID="1cd181aa1c7877618b10feebf263eb77128eb8ddc06045a713ee6fb54ee067a4"
Jul 17 13:20:57.516460 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:20:57.516428    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh_odh-ai-gateway-infra(e9d0a31c-6a77-466f-b083-397ead8621b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh" podUID="e9d0a31c-6a77-466f-b083-397ead8621b8"
Jul 17 13:20:57.521135 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:57.521118    2569 scope.go:117] "RemoveContainer" containerID="7af5ee24c6f63837c2e0bb4c17523cbdaddd6a85039b19f052b4d2d71502a217"
Jul 17 13:20:57.547626 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:57.547606    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x"]
Jul 17 13:20:57.553792 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:57.553771    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-h9b7x"]
Jul 17 13:20:57.634209 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:57.634189    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e7880e8-010b-4e9f-98b1-c4be084947c7" path="/var/lib/kubelet/pods/9e7880e8-010b-4e9f-98b1-c4be084947c7/volumes"
Jul 17 13:20:57.634555 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:57.634541    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd" path="/var/lib/kubelet/pods/e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd/volumes"
Jul 17 13:20:58.543483 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:58.543452    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh"]
Jul 17 13:20:58.683145 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:58.683122    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh"
Jul 17 13:20:58.736403 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:58.736375    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0a31c-6a77-466f-b083-397ead8621b8-maas-api-service-ca\") pod \"e9d0a31c-6a77-466f-b083-397ead8621b8\" (UID: \"e9d0a31c-6a77-466f-b083-397ead8621b8\") "
Jul 17 13:20:58.736688 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:58.736669    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d0a31c-6a77-466f-b083-397ead8621b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9d0a31c-6a77-466f-b083-397ead8621b8" (UID: "e9d0a31c-6a77-466f-b083-397ead8621b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:20:58.837023 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:58.836958    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0a31c-6a77-466f-b083-397ead8621b8-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:20:59.168686 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:59.168613    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:20:59.528814 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:59.528791    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh"
Jul 17 13:20:59.528970 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:59.528828    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh" event={"ID":"e9d0a31c-6a77-466f-b083-397ead8621b8","Type":"ContainerDied","Data":"cc2138e5eea23163153200d3a979b42cfecfb53ec11f1ec6f2a3151cf6edc389"}
Jul 17 13:20:59.528970 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:59.528895    2569 scope.go:117] "RemoveContainer" containerID="1cd181aa1c7877618b10feebf263eb77128eb8ddc06045a713ee6fb54ee067a4"
Jul 17 13:20:59.550257 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:59.550235    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh"]
Jul 17 13:20:59.556201 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:59.556183    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-ng6kh"]
Jul 17 13:20:59.634562 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:20:59.634539    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9d0a31c-6a77-466f-b083-397ead8621b8" path="/var/lib/kubelet/pods/e9d0a31c-6a77-466f-b083-397ead8621b8/volumes"
Jul 17 13:21:10.629096 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:10.629068    2569 scope.go:117] "RemoveContainer" containerID="ee9ead00dc05a65f122a195823a047e044d42f420ff503c2405b2a59d3d22516"
Jul 17 13:21:11.585518 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:11.585489    2569 generic.go:356] "Generic (PLEG): container finished" podID="9c948e35-18b6-454f-9167-c743fbf431d3" containerID="cb5e846bc7b730216b52db4e7c4410eb34b6e4b66e54e101b6a9aec1f27b9c9d" exitCode=6
Jul 17 13:21:11.585642 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:11.585562    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq" event={"ID":"9c948e35-18b6-454f-9167-c743fbf431d3","Type":"ContainerDied","Data":"cb5e846bc7b730216b52db4e7c4410eb34b6e4b66e54e101b6a9aec1f27b9c9d"}
Jul 17 13:21:11.585642 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:11.585627    2569 scope.go:117] "RemoveContainer" containerID="ee9ead00dc05a65f122a195823a047e044d42f420ff503c2405b2a59d3d22516"
Jul 17 13:21:11.585972 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:11.585947    2569 scope.go:117] "RemoveContainer" containerID="cb5e846bc7b730216b52db4e7c4410eb34b6e4b66e54e101b6a9aec1f27b9c9d"
Jul 17 13:21:11.586201 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:21:11.586179    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq_odh-ai-gateway-infra(9c948e35-18b6-454f-9167-c743fbf431d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq" podUID="9c948e35-18b6-454f-9167-c743fbf431d3"
Jul 17 13:21:12.611766 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:12.611719    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq"]
Jul 17 13:21:12.737613 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:12.737590    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq"
Jul 17 13:21:12.841542 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:12.841516    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c948e35-18b6-454f-9167-c743fbf431d3-maas-api-service-ca\") pod \"9c948e35-18b6-454f-9167-c743fbf431d3\" (UID: \"9c948e35-18b6-454f-9167-c743fbf431d3\") "
Jul 17 13:21:12.841828 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:12.841808    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c948e35-18b6-454f-9167-c743fbf431d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9c948e35-18b6-454f-9167-c743fbf431d3" (UID: "9c948e35-18b6-454f-9167-c743fbf431d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:21:12.942765 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:12.942688    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c948e35-18b6-454f-9167-c743fbf431d3-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:21:13.596953 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:13.596931    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq"
Jul 17 13:21:13.597087 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:13.596926    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq" event={"ID":"9c948e35-18b6-454f-9167-c743fbf431d3","Type":"ContainerDied","Data":"1a6f557a8a89c5bd5848a465f3c5142cc9029f480dccb7c09856f5cbc2f871a1"}
Jul 17 13:21:13.597087 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:13.597069    2569 scope.go:117] "RemoveContainer" containerID="cb5e846bc7b730216b52db4e7c4410eb34b6e4b66e54e101b6a9aec1f27b9c9d"
Jul 17 13:21:13.621305 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:13.621282    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq"]
Jul 17 13:21:13.633797 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:13.633780    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-w6knq"]
Jul 17 13:21:15.634291 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:15.634257    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c948e35-18b6-454f-9167-c743fbf431d3" path="/var/lib/kubelet/pods/9c948e35-18b6-454f-9167-c743fbf431d3/volumes"
Jul 17 13:21:28.679203 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679169    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2"]
Jul 17 13:21:28.679594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679494    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d0a31c-6a77-466f-b083-397ead8621b8" containerName="revoke-keys"
Jul 17 13:21:28.679594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679506    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d0a31c-6a77-466f-b083-397ead8621b8" containerName="revoke-keys"
Jul 17 13:21:28.679594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679514    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c948e35-18b6-454f-9167-c743fbf431d3" containerName="revoke-keys"
Jul 17 13:21:28.679594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679520    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c948e35-18b6-454f-9167-c743fbf431d3" containerName="revoke-keys"
Jul 17 13:21:28.679594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679535    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e7880e8-010b-4e9f-98b1-c4be084947c7" containerName="revoke-keys"
Jul 17 13:21:28.679594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679540    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e7880e8-010b-4e9f-98b1-c4be084947c7" containerName="revoke-keys"
Jul 17 13:21:28.679594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679547    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e7880e8-010b-4e9f-98b1-c4be084947c7" containerName="revoke-keys"
Jul 17 13:21:28.679594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679552    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e7880e8-010b-4e9f-98b1-c4be084947c7" containerName="revoke-keys"
Jul 17 13:21:28.679594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679564    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd" containerName="maas-api"
Jul 17 13:21:28.679594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679572    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd" containerName="maas-api"
Jul 17 13:21:28.679594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679580    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d0a31c-6a77-466f-b083-397ead8621b8" containerName="revoke-keys"
Jul 17 13:21:28.679594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679584    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d0a31c-6a77-466f-b083-397ead8621b8" containerName="revoke-keys"
Jul 17 13:21:28.679594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679599    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c948e35-18b6-454f-9167-c743fbf431d3" containerName="revoke-keys"
Jul 17 13:21:28.680026 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679604    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c948e35-18b6-454f-9167-c743fbf431d3" containerName="revoke-keys"
Jul 17 13:21:28.680026 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679659    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0e7ed7a-cc4b-4b71-a3c6-1ad4e1b8e6fd" containerName="maas-api"
Jul 17 13:21:28.680026 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679667    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c948e35-18b6-454f-9167-c743fbf431d3" containerName="revoke-keys"
Jul 17 13:21:28.680026 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679673    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e7880e8-010b-4e9f-98b1-c4be084947c7" containerName="revoke-keys"
Jul 17 13:21:28.680026 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679677    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d0a31c-6a77-466f-b083-397ead8621b8" containerName="revoke-keys"
Jul 17 13:21:28.680026 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679683    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c948e35-18b6-454f-9167-c743fbf431d3" containerName="revoke-keys"
Jul 17 13:21:28.680026 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679688    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d0a31c-6a77-466f-b083-397ead8621b8" containerName="revoke-keys"
Jul 17 13:21:28.680026 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.679693    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e7880e8-010b-4e9f-98b1-c4be084947c7" containerName="revoke-keys"
Jul 17 13:21:28.682681 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.682658    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2"
Jul 17 13:21:28.685783 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.685760    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:21:28.685874 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.685855    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:21:28.691480 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.691458    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2"]
Jul 17 13:21:28.763506 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.763480    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc90b01f-0737-4c85-b55e-7be7e0d63174-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2\" (UID: \"dc90b01f-0737-4c85-b55e-7be7e0d63174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2"
Jul 17 13:21:28.864033 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.864009    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc90b01f-0737-4c85-b55e-7be7e0d63174-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2\" (UID: \"dc90b01f-0737-4c85-b55e-7be7e0d63174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2"
Jul 17 13:21:28.864537 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.864520    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc90b01f-0737-4c85-b55e-7be7e0d63174-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2\" (UID: \"dc90b01f-0737-4c85-b55e-7be7e0d63174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2"
Jul 17 13:21:28.994986 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:28.994963    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2"
Jul 17 13:21:29.332716 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:29.332589    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2"]
Jul 17 13:21:29.335014 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:21:29.334986    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc90b01f_0737_4c85_b55e_7be7e0d63174.slice/crio-7f7a40a8e4c1d5c551cf2589adab8f470fb578db4c0bb8430632b1e3e6fcd3fd WatchSource:0}: Error finding container 7f7a40a8e4c1d5c551cf2589adab8f470fb578db4c0bb8430632b1e3e6fcd3fd: Status 404 returned error can't find the container with id 7f7a40a8e4c1d5c551cf2589adab8f470fb578db4c0bb8430632b1e3e6fcd3fd
Jul 17 13:21:29.678095 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:29.678032    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2" event={"ID":"dc90b01f-0737-4c85-b55e-7be7e0d63174","Type":"ContainerStarted","Data":"2d68b996d4002512134aa4b5d213635a752d4f6c64289233b95494434ce889d2"}
Jul 17 13:21:29.678095 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:29.678079    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2" event={"ID":"dc90b01f-0737-4c85-b55e-7be7e0d63174","Type":"ContainerStarted","Data":"7f7a40a8e4c1d5c551cf2589adab8f470fb578db4c0bb8430632b1e3e6fcd3fd"}
Jul 17 13:21:29.698699 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:29.698656    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2" podStartSLOduration=1.698641629 podStartE2EDuration="1.698641629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:21:29.697234804 +0000 UTC m=+3564.701281687" watchObservedRunningTime="2026-07-17 13:21:29.698641629 +0000 UTC m=+3564.702688510"
Jul 17 13:21:30.685260 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:30.685226    2569 generic.go:356] "Generic (PLEG): container finished" podID="dc90b01f-0737-4c85-b55e-7be7e0d63174" containerID="2d68b996d4002512134aa4b5d213635a752d4f6c64289233b95494434ce889d2" exitCode=6
Jul 17 13:21:30.685459 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:30.685327    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2" event={"ID":"dc90b01f-0737-4c85-b55e-7be7e0d63174","Type":"ContainerDied","Data":"2d68b996d4002512134aa4b5d213635a752d4f6c64289233b95494434ce889d2"}
Jul 17 13:21:30.685522 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:30.685477    2569 scope.go:117] "RemoveContainer" containerID="2d68b996d4002512134aa4b5d213635a752d4f6c64289233b95494434ce889d2"
Jul 17 13:21:31.690661 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:31.690636    2569 generic.go:356] "Generic (PLEG): container finished" podID="dc90b01f-0737-4c85-b55e-7be7e0d63174" containerID="593c8877fd838c5c838bc436af2caf74736f77bfc2c3457c715930f95b4e271c" exitCode=6
Jul 17 13:21:31.691038 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:31.690724    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2" event={"ID":"dc90b01f-0737-4c85-b55e-7be7e0d63174","Type":"ContainerDied","Data":"593c8877fd838c5c838bc436af2caf74736f77bfc2c3457c715930f95b4e271c"}
Jul 17 13:21:31.691038 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:31.690795    2569 scope.go:117] "RemoveContainer" containerID="2d68b996d4002512134aa4b5d213635a752d4f6c64289233b95494434ce889d2"
Jul 17 13:21:31.691038 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:31.690851    2569 scope.go:117] "RemoveContainer" containerID="593c8877fd838c5c838bc436af2caf74736f77bfc2c3457c715930f95b4e271c"
Jul 17 13:21:31.691038 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:21:31.691000    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2_odh-ai-gateway-infra(dc90b01f-0737-4c85-b55e-7be7e0d63174)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2" podUID="dc90b01f-0737-4c85-b55e-7be7e0d63174"
Jul 17 13:21:32.697856 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:32.697828    2569 scope.go:117] "RemoveContainer" containerID="593c8877fd838c5c838bc436af2caf74736f77bfc2c3457c715930f95b4e271c"
Jul 17 13:21:32.698248 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:21:32.697997    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2_odh-ai-gateway-infra(dc90b01f-0737-4c85-b55e-7be7e0d63174)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2" podUID="dc90b01f-0737-4c85-b55e-7be7e0d63174"
Jul 17 13:21:44.628484 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:44.628454    2569 scope.go:117] "RemoveContainer" containerID="593c8877fd838c5c838bc436af2caf74736f77bfc2c3457c715930f95b4e271c"
Jul 17 13:21:45.756252 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:45.756219    2569 generic.go:356] "Generic (PLEG): container finished" podID="dc90b01f-0737-4c85-b55e-7be7e0d63174" containerID="af5fff278d1e6b9a5ad20d979f6f46d51805456ed7cc48f449bedd0c9352cd46" exitCode=6
Jul 17 13:21:45.756686 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:45.756276    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2" event={"ID":"dc90b01f-0737-4c85-b55e-7be7e0d63174","Type":"ContainerDied","Data":"af5fff278d1e6b9a5ad20d979f6f46d51805456ed7cc48f449bedd0c9352cd46"}
Jul 17 13:21:45.756686 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:45.756325    2569 scope.go:117] "RemoveContainer" containerID="593c8877fd838c5c838bc436af2caf74736f77bfc2c3457c715930f95b4e271c"
Jul 17 13:21:45.756686 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:45.756575    2569 scope.go:117] "RemoveContainer" containerID="af5fff278d1e6b9a5ad20d979f6f46d51805456ed7cc48f449bedd0c9352cd46"
Jul 17 13:21:45.756892 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:21:45.756784    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2_odh-ai-gateway-infra(dc90b01f-0737-4c85-b55e-7be7e0d63174)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2" podUID="dc90b01f-0737-4c85-b55e-7be7e0d63174"
Jul 17 13:21:46.789771 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:46.789726    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2"]
Jul 17 13:21:46.918940 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:46.918914    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2"
Jul 17 13:21:46.995354 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:46.995326    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc90b01f-0737-4c85-b55e-7be7e0d63174-maas-api-service-ca\") pod \"dc90b01f-0737-4c85-b55e-7be7e0d63174\" (UID: \"dc90b01f-0737-4c85-b55e-7be7e0d63174\") "
Jul 17 13:21:46.995646 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:46.995625    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc90b01f-0737-4c85-b55e-7be7e0d63174-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc90b01f-0737-4c85-b55e-7be7e0d63174" (UID: "dc90b01f-0737-4c85-b55e-7be7e0d63174"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:21:47.096057 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:47.095993    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc90b01f-0737-4c85-b55e-7be7e0d63174-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:21:47.768179 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:47.768157    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2"
Jul 17 13:21:47.768179 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:47.768159    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2" event={"ID":"dc90b01f-0737-4c85-b55e-7be7e0d63174","Type":"ContainerDied","Data":"7f7a40a8e4c1d5c551cf2589adab8f470fb578db4c0bb8430632b1e3e6fcd3fd"}
Jul 17 13:21:47.768379 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:47.768215    2569 scope.go:117] "RemoveContainer" containerID="af5fff278d1e6b9a5ad20d979f6f46d51805456ed7cc48f449bedd0c9352cd46"
Jul 17 13:21:47.786378 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:47.786353    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2"]
Jul 17 13:21:47.791010 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:47.790987    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-gj7x2"]
Jul 17 13:21:49.633514 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:49.633472    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc90b01f-0737-4c85-b55e-7be7e0d63174" path="/var/lib/kubelet/pods/dc90b01f-0737-4c85-b55e-7be7e0d63174/volumes"
Jul 17 13:21:52.352746 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.352714    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9"]
Jul 17 13:21:52.353695 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.353671    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c948e35-18b6-454f-9167-c743fbf431d3" containerName="revoke-keys"
Jul 17 13:21:52.353856 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.353844    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c948e35-18b6-454f-9167-c743fbf431d3" containerName="revoke-keys"
Jul 17 13:21:52.353951 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.353941    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d0a31c-6a77-466f-b083-397ead8621b8" containerName="revoke-keys"
Jul 17 13:21:52.354028 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.354018    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d0a31c-6a77-466f-b083-397ead8621b8" containerName="revoke-keys"
Jul 17 13:21:52.354104 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.354096    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc90b01f-0737-4c85-b55e-7be7e0d63174" containerName="revoke-keys"
Jul 17 13:21:52.354171 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.354163    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc90b01f-0737-4c85-b55e-7be7e0d63174" containerName="revoke-keys"
Jul 17 13:21:52.354246 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.354238    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc90b01f-0737-4c85-b55e-7be7e0d63174" containerName="revoke-keys"
Jul 17 13:21:52.354316 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.354308    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc90b01f-0737-4c85-b55e-7be7e0d63174" containerName="revoke-keys"
Jul 17 13:21:52.354395 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.354387    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e7880e8-010b-4e9f-98b1-c4be084947c7" containerName="revoke-keys"
Jul 17 13:21:52.354467 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.354459    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e7880e8-010b-4e9f-98b1-c4be084947c7" containerName="revoke-keys"
Jul 17 13:21:52.354622 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.354612    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c948e35-18b6-454f-9167-c743fbf431d3" containerName="revoke-keys"
Jul 17 13:21:52.354707 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.354698    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc90b01f-0737-4c85-b55e-7be7e0d63174" containerName="revoke-keys"
Jul 17 13:21:52.354797 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.354788    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d0a31c-6a77-466f-b083-397ead8621b8" containerName="revoke-keys"
Jul 17 13:21:52.354884 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.354875    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e7880e8-010b-4e9f-98b1-c4be084947c7" containerName="revoke-keys"
Jul 17 13:21:52.354960 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.354952    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc90b01f-0737-4c85-b55e-7be7e0d63174" containerName="revoke-keys"
Jul 17 13:21:52.358921 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.358895    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9"
Jul 17 13:21:52.362119 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.362099    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:21:52.362425 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.362412    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:21:52.365815 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.365795    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9"]
Jul 17 13:21:52.439311 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.439282    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec7dedd-ea76-4132-89e8-557d990446f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9\" (UID: \"0ec7dedd-ea76-4132-89e8-557d990446f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9"
Jul 17 13:21:52.540629 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.540601    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec7dedd-ea76-4132-89e8-557d990446f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9\" (UID: \"0ec7dedd-ea76-4132-89e8-557d990446f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9"
Jul 17 13:21:52.541214 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.541194    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec7dedd-ea76-4132-89e8-557d990446f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9\" (UID: \"0ec7dedd-ea76-4132-89e8-557d990446f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9"
Jul 17 13:21:52.673207 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.673155    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9"
Jul 17 13:21:52.801453 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:52.801429    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9"]
Jul 17 13:21:52.803005 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:21:52.802978    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec7dedd_ea76_4132_89e8_557d990446f0.slice/crio-bb6a67d904023ab5dbc476b78432174c659dae191ad758db533b3a78c9c77d67 WatchSource:0}: Error finding container bb6a67d904023ab5dbc476b78432174c659dae191ad758db533b3a78c9c77d67: Status 404 returned error can't find the container with id bb6a67d904023ab5dbc476b78432174c659dae191ad758db533b3a78c9c77d67
Jul 17 13:21:53.797656 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:53.797583    2569 generic.go:356] "Generic (PLEG): container finished" podID="0ec7dedd-ea76-4132-89e8-557d990446f0" containerID="076155ba28630898943ee1f44ffe6b410e9bccaa8b45c258230d80e257527f25" exitCode=6
Jul 17 13:21:53.797656 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:53.797645    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9" event={"ID":"0ec7dedd-ea76-4132-89e8-557d990446f0","Type":"ContainerDied","Data":"076155ba28630898943ee1f44ffe6b410e9bccaa8b45c258230d80e257527f25"}
Jul 17 13:21:53.798088 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:53.797681    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9" event={"ID":"0ec7dedd-ea76-4132-89e8-557d990446f0","Type":"ContainerStarted","Data":"bb6a67d904023ab5dbc476b78432174c659dae191ad758db533b3a78c9c77d67"}
Jul 17 13:21:53.798088 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:53.797874    2569 scope.go:117] "RemoveContainer" containerID="076155ba28630898943ee1f44ffe6b410e9bccaa8b45c258230d80e257527f25"
Jul 17 13:21:54.803837 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:54.803800    2569 generic.go:356] "Generic (PLEG): container finished" podID="0ec7dedd-ea76-4132-89e8-557d990446f0" containerID="da09631e54bd06b9099ee0b56e76030d46141a80843cee196de1da3546aeb390" exitCode=6
Jul 17 13:21:54.804203 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:54.803887    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9" event={"ID":"0ec7dedd-ea76-4132-89e8-557d990446f0","Type":"ContainerDied","Data":"da09631e54bd06b9099ee0b56e76030d46141a80843cee196de1da3546aeb390"}
Jul 17 13:21:54.804203 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:54.803942    2569 scope.go:117] "RemoveContainer" containerID="076155ba28630898943ee1f44ffe6b410e9bccaa8b45c258230d80e257527f25"
Jul 17 13:21:54.804203 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:54.804090    2569 scope.go:117] "RemoveContainer" containerID="da09631e54bd06b9099ee0b56e76030d46141a80843cee196de1da3546aeb390"
Jul 17 13:21:54.804346 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:21:54.804312    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9_odh-ai-gateway-infra(0ec7dedd-ea76-4132-89e8-557d990446f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9" podUID="0ec7dedd-ea76-4132-89e8-557d990446f0"
Jul 17 13:21:55.809794 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:55.809768    2569 scope.go:117] "RemoveContainer" containerID="da09631e54bd06b9099ee0b56e76030d46141a80843cee196de1da3546aeb390"
Jul 17 13:21:55.810162 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:21:55.809934    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9_odh-ai-gateway-infra(0ec7dedd-ea76-4132-89e8-557d990446f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9" podUID="0ec7dedd-ea76-4132-89e8-557d990446f0"
Jul 17 13:21:56.488966 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:21:56.488939    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:22:08.628270 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:08.628234    2569 scope.go:117] "RemoveContainer" containerID="da09631e54bd06b9099ee0b56e76030d46141a80843cee196de1da3546aeb390"
Jul 17 13:22:08.870169 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:08.870141    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9" event={"ID":"0ec7dedd-ea76-4132-89e8-557d990446f0","Type":"ContainerStarted","Data":"ae2c709f62448617e8e4c895da63f20c11cbab6572071ccda76aa3d30809ab9e"}
Jul 17 13:22:08.892575 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:08.892505    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9" podStartSLOduration=16.89249183 podStartE2EDuration="16.89249183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:22:08.890111569 +0000 UTC m=+3603.894158451" watchObservedRunningTime="2026-07-17 13:22:08.89249183 +0000 UTC m=+3603.896538712"
Jul 17 13:22:09.727937 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:09.727904    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:22:09.876798 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:09.876740    2569 generic.go:356] "Generic (PLEG): container finished" podID="0ec7dedd-ea76-4132-89e8-557d990446f0" containerID="ae2c709f62448617e8e4c895da63f20c11cbab6572071ccda76aa3d30809ab9e" exitCode=6
Jul 17 13:22:09.876960 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:09.876803    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9" event={"ID":"0ec7dedd-ea76-4132-89e8-557d990446f0","Type":"ContainerDied","Data":"ae2c709f62448617e8e4c895da63f20c11cbab6572071ccda76aa3d30809ab9e"}
Jul 17 13:22:09.876960 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:09.876869    2569 scope.go:117] "RemoveContainer" containerID="da09631e54bd06b9099ee0b56e76030d46141a80843cee196de1da3546aeb390"
Jul 17 13:22:09.877124 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:09.877106    2569 scope.go:117] "RemoveContainer" containerID="ae2c709f62448617e8e4c895da63f20c11cbab6572071ccda76aa3d30809ab9e"
Jul 17 13:22:09.877330 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:22:09.877306    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9_odh-ai-gateway-infra(0ec7dedd-ea76-4132-89e8-557d990446f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9" podUID="0ec7dedd-ea76-4132-89e8-557d990446f0"
Jul 17 13:22:09.918370 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:09.918344    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9"]
Jul 17 13:22:11.011213 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:11.011192    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9"
Jul 17 13:22:11.075263 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:11.075234    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec7dedd-ea76-4132-89e8-557d990446f0-maas-api-service-ca\") pod \"0ec7dedd-ea76-4132-89e8-557d990446f0\" (UID: \"0ec7dedd-ea76-4132-89e8-557d990446f0\") "
Jul 17 13:22:11.075583 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:11.075564    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec7dedd-ea76-4132-89e8-557d990446f0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ec7dedd-ea76-4132-89e8-557d990446f0" (UID: "0ec7dedd-ea76-4132-89e8-557d990446f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:22:11.176099 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:11.176042    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec7dedd-ea76-4132-89e8-557d990446f0-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:22:11.888974 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:11.888951    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9"
Jul 17 13:22:11.889091 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:11.888948    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9" event={"ID":"0ec7dedd-ea76-4132-89e8-557d990446f0","Type":"ContainerDied","Data":"bb6a67d904023ab5dbc476b78432174c659dae191ad758db533b3a78c9c77d67"}
Jul 17 13:22:11.889149 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:11.889092    2569 scope.go:117] "RemoveContainer" containerID="ae2c709f62448617e8e4c895da63f20c11cbab6572071ccda76aa3d30809ab9e"
Jul 17 13:22:11.907908 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:11.907878    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9"]
Jul 17 13:22:11.912942 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:11.912918    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-lwbn9"]
Jul 17 13:22:13.491486 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:13.491456    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 13:22:13.504686 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:13.498143    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 13:22:13.634054 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:13.634020    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ec7dedd-ea76-4132-89e8-557d990446f0" path="/var/lib/kubelet/pods/0ec7dedd-ea76-4132-89e8-557d990446f0/volumes"
Jul 17 13:22:18.726040 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:18.726002    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9"]
Jul 17 13:22:18.726520 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:18.726355    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ec7dedd-ea76-4132-89e8-557d990446f0" containerName="revoke-keys"
Jul 17 13:22:18.726520 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:18.726367    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ec7dedd-ea76-4132-89e8-557d990446f0" containerName="revoke-keys"
Jul 17 13:22:18.726520 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:18.726382    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc90b01f-0737-4c85-b55e-7be7e0d63174" containerName="revoke-keys"
Jul 17 13:22:18.726520 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:18.726388    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc90b01f-0737-4c85-b55e-7be7e0d63174" containerName="revoke-keys"
Jul 17 13:22:18.726520 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:18.726407    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ec7dedd-ea76-4132-89e8-557d990446f0" containerName="revoke-keys"
Jul 17 13:22:18.726520 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:18.726413    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ec7dedd-ea76-4132-89e8-557d990446f0" containerName="revoke-keys"
Jul 17 13:22:18.726520 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:18.726465    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ec7dedd-ea76-4132-89e8-557d990446f0" containerName="revoke-keys"
Jul 17 13:22:18.726520 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:18.726472    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ec7dedd-ea76-4132-89e8-557d990446f0" containerName="revoke-keys"
Jul 17 13:22:18.726520 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:18.726479    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc90b01f-0737-4c85-b55e-7be7e0d63174" containerName="revoke-keys"
Jul 17 13:22:18.726520 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:18.726486    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ec7dedd-ea76-4132-89e8-557d990446f0" containerName="revoke-keys"
Jul 17 13:22:18.729766 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:18.729727    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9"
Jul 17 13:22:18.732932 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:18.732911    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:22:18.733279 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:18.733258    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:22:18.740607 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:18.740582    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9"]
Jul 17 13:22:18.829993 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:18.829966    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/330fef4b-5bd2-4dc2-a56f-f73350b8b29a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9\" (UID: \"330fef4b-5bd2-4dc2-a56f-f73350b8b29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9"
Jul 17 13:22:18.931149 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:18.931118    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/330fef4b-5bd2-4dc2-a56f-f73350b8b29a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9\" (UID: \"330fef4b-5bd2-4dc2-a56f-f73350b8b29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9"
Jul 17 13:22:18.931746 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:18.931725    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/330fef4b-5bd2-4dc2-a56f-f73350b8b29a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9\" (UID: \"330fef4b-5bd2-4dc2-a56f-f73350b8b29a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9"
Jul 17 13:22:19.041484 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:19.041426    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9"
Jul 17 13:22:19.190620 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:19.190488    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9"]
Jul 17 13:22:19.193174 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:22:19.193139    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330fef4b_5bd2_4dc2_a56f_f73350b8b29a.slice/crio-6a8bf16592e1d1627a1c9e9b4f47f9bdd717e7b5d74982d4b09024c0e833e428 WatchSource:0}: Error finding container 6a8bf16592e1d1627a1c9e9b4f47f9bdd717e7b5d74982d4b09024c0e833e428: Status 404 returned error can't find the container with id 6a8bf16592e1d1627a1c9e9b4f47f9bdd717e7b5d74982d4b09024c0e833e428
Jul 17 13:22:19.931945 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:19.931916    2569 generic.go:356] "Generic (PLEG): container finished" podID="330fef4b-5bd2-4dc2-a56f-f73350b8b29a" containerID="fd582d884f6d3b161e625c89db9ab7df7b67121b80028c02791e6e93d8c5cc50" exitCode=6
Jul 17 13:22:19.932232 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:19.931977    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9" event={"ID":"330fef4b-5bd2-4dc2-a56f-f73350b8b29a","Type":"ContainerDied","Data":"fd582d884f6d3b161e625c89db9ab7df7b67121b80028c02791e6e93d8c5cc50"}
Jul 17 13:22:19.932232 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:19.932030    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9" event={"ID":"330fef4b-5bd2-4dc2-a56f-f73350b8b29a","Type":"ContainerStarted","Data":"6a8bf16592e1d1627a1c9e9b4f47f9bdd717e7b5d74982d4b09024c0e833e428"}
Jul 17 13:22:19.932232 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:19.932188    2569 scope.go:117] "RemoveContainer" containerID="fd582d884f6d3b161e625c89db9ab7df7b67121b80028c02791e6e93d8c5cc50"
Jul 17 13:22:20.940660 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:20.940586    2569 generic.go:356] "Generic (PLEG): container finished" podID="330fef4b-5bd2-4dc2-a56f-f73350b8b29a" containerID="9d13cb7d236056f4a95a489949453fb339e337435a9c72f84cdfd7747047fb10" exitCode=6
Jul 17 13:22:20.941112 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:20.940658    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9" event={"ID":"330fef4b-5bd2-4dc2-a56f-f73350b8b29a","Type":"ContainerDied","Data":"9d13cb7d236056f4a95a489949453fb339e337435a9c72f84cdfd7747047fb10"}
Jul 17 13:22:20.941112 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:20.940711    2569 scope.go:117] "RemoveContainer" containerID="fd582d884f6d3b161e625c89db9ab7df7b67121b80028c02791e6e93d8c5cc50"
Jul 17 13:22:20.941112 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:20.940905    2569 scope.go:117] "RemoveContainer" containerID="9d13cb7d236056f4a95a489949453fb339e337435a9c72f84cdfd7747047fb10"
Jul 17 13:22:20.941288 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:22:20.941110    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9_odh-ai-gateway-infra(330fef4b-5bd2-4dc2-a56f-f73350b8b29a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9" podUID="330fef4b-5bd2-4dc2-a56f-f73350b8b29a"
Jul 17 13:22:21.947065 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:21.947026    2569 scope.go:117] "RemoveContainer" containerID="9d13cb7d236056f4a95a489949453fb339e337435a9c72f84cdfd7747047fb10"
Jul 17 13:22:21.947453 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:22:21.947196    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9_odh-ai-gateway-infra(330fef4b-5bd2-4dc2-a56f-f73350b8b29a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9" podUID="330fef4b-5bd2-4dc2-a56f-f73350b8b29a"
Jul 17 13:22:30.050601 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:30.050558    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz"]
Jul 17 13:22:30.051227 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:30.051101    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ec7dedd-ea76-4132-89e8-557d990446f0" containerName="revoke-keys"
Jul 17 13:22:30.051227 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:30.051122    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ec7dedd-ea76-4132-89e8-557d990446f0" containerName="revoke-keys"
Jul 17 13:22:30.055226 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:30.055202    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz"
Jul 17 13:22:30.070674 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:30.070647    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz"]
Jul 17 13:22:30.115564 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:30.115538    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75687e5-6e3b-4363-843c-280111c2a143-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz\" (UID: \"a75687e5-6e3b-4363-843c-280111c2a143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz"
Jul 17 13:22:30.216680 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:30.216651    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75687e5-6e3b-4363-843c-280111c2a143-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz\" (UID: \"a75687e5-6e3b-4363-843c-280111c2a143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz"
Jul 17 13:22:30.217331 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:30.217308    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75687e5-6e3b-4363-843c-280111c2a143-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz\" (UID: \"a75687e5-6e3b-4363-843c-280111c2a143\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz"
Jul 17 13:22:30.366239 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:30.366178    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz"
Jul 17 13:22:30.718855 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:30.718831    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz"]
Jul 17 13:22:30.721095 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:22:30.721069    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75687e5_6e3b_4363_843c_280111c2a143.slice/crio-328fdb0e7b202c734c99f974aa132b9cf8658ee9aba1073d8c628b6634ac80ac WatchSource:0}: Error finding container 328fdb0e7b202c734c99f974aa132b9cf8658ee9aba1073d8c628b6634ac80ac: Status 404 returned error can't find the container with id 328fdb0e7b202c734c99f974aa132b9cf8658ee9aba1073d8c628b6634ac80ac
Jul 17 13:22:30.996811 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:30.996784    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz" event={"ID":"a75687e5-6e3b-4363-843c-280111c2a143","Type":"ContainerStarted","Data":"328fdb0e7b202c734c99f974aa132b9cf8658ee9aba1073d8c628b6634ac80ac"}
Jul 17 13:22:32.003495 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:32.003458    2569 generic.go:356] "Generic (PLEG): container finished" podID="a75687e5-6e3b-4363-843c-280111c2a143" containerID="8475deb878854588f6ed06f4b4f91bdb9a460765c107da4bdf5a1f07412cc994" exitCode=6
Jul 17 13:22:32.003919 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:32.003554    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz" event={"ID":"a75687e5-6e3b-4363-843c-280111c2a143","Type":"ContainerDied","Data":"8475deb878854588f6ed06f4b4f91bdb9a460765c107da4bdf5a1f07412cc994"}
Jul 17 13:22:32.003919 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:32.003710    2569 scope.go:117] "RemoveContainer" containerID="8475deb878854588f6ed06f4b4f91bdb9a460765c107da4bdf5a1f07412cc994"
Jul 17 13:22:33.010048 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:33.010020    2569 generic.go:356] "Generic (PLEG): container finished" podID="a75687e5-6e3b-4363-843c-280111c2a143" containerID="64895264a7ff3a1276aad1ae92d2bb91cb4e26fcbd03b465b76142f46a1c829f" exitCode=6
Jul 17 13:22:33.010407 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:33.010056    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz" event={"ID":"a75687e5-6e3b-4363-843c-280111c2a143","Type":"ContainerDied","Data":"64895264a7ff3a1276aad1ae92d2bb91cb4e26fcbd03b465b76142f46a1c829f"}
Jul 17 13:22:33.010407 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:33.010108    2569 scope.go:117] "RemoveContainer" containerID="8475deb878854588f6ed06f4b4f91bdb9a460765c107da4bdf5a1f07412cc994"
Jul 17 13:22:33.010407 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:33.010263    2569 scope.go:117] "RemoveContainer" containerID="64895264a7ff3a1276aad1ae92d2bb91cb4e26fcbd03b465b76142f46a1c829f"
Jul 17 13:22:33.010532 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:22:33.010458    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz_odh-ai-gateway-infra(a75687e5-6e3b-4363-843c-280111c2a143)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz" podUID="a75687e5-6e3b-4363-843c-280111c2a143"
Jul 17 13:22:34.016014 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:34.015987    2569 scope.go:117] "RemoveContainer" containerID="64895264a7ff3a1276aad1ae92d2bb91cb4e26fcbd03b465b76142f46a1c829f"
Jul 17 13:22:34.016427 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:22:34.016157    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz_odh-ai-gateway-infra(a75687e5-6e3b-4363-843c-280111c2a143)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz" podUID="a75687e5-6e3b-4363-843c-280111c2a143"
Jul 17 13:22:35.630124 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:35.630090    2569 scope.go:117] "RemoveContainer" containerID="9d13cb7d236056f4a95a489949453fb339e337435a9c72f84cdfd7747047fb10"
Jul 17 13:22:36.026326 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:36.026294    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9" event={"ID":"330fef4b-5bd2-4dc2-a56f-f73350b8b29a","Type":"ContainerStarted","Data":"8cc4e1fac787a963b7abc5ae93a8e774fce4505202d7bc707ca9d2624a394783"}
Jul 17 13:22:36.057406 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:36.057305    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9" podStartSLOduration=18.057290265 podStartE2EDuration="18.057290265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:22:36.056702571 +0000 UTC m=+3631.060749454" watchObservedRunningTime="2026-07-17 13:22:36.057290265 +0000 UTC m=+3631.061337148"
Jul 17 13:22:37.033289 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:37.033252    2569 generic.go:356] "Generic (PLEG): container finished" podID="330fef4b-5bd2-4dc2-a56f-f73350b8b29a" containerID="8cc4e1fac787a963b7abc5ae93a8e774fce4505202d7bc707ca9d2624a394783" exitCode=6
Jul 17 13:22:37.033731 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:37.033325    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9" event={"ID":"330fef4b-5bd2-4dc2-a56f-f73350b8b29a","Type":"ContainerDied","Data":"8cc4e1fac787a963b7abc5ae93a8e774fce4505202d7bc707ca9d2624a394783"}
Jul 17 13:22:37.033731 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:37.033391    2569 scope.go:117] "RemoveContainer" containerID="9d13cb7d236056f4a95a489949453fb339e337435a9c72f84cdfd7747047fb10"
Jul 17 13:22:37.033731 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:37.033699    2569 scope.go:117] "RemoveContainer" containerID="8cc4e1fac787a963b7abc5ae93a8e774fce4505202d7bc707ca9d2624a394783"
Jul 17 13:22:37.033968 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:22:37.033942    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9_odh-ai-gateway-infra(330fef4b-5bd2-4dc2-a56f-f73350b8b29a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9" podUID="330fef4b-5bd2-4dc2-a56f-f73350b8b29a"
Jul 17 13:22:37.067087 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:37.067062    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9"]
Jul 17 13:22:38.179061 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:38.179039    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9"
Jul 17 13:22:38.281587 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:38.281568    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/330fef4b-5bd2-4dc2-a56f-f73350b8b29a-maas-api-service-ca\") pod \"330fef4b-5bd2-4dc2-a56f-f73350b8b29a\" (UID: \"330fef4b-5bd2-4dc2-a56f-f73350b8b29a\") "
Jul 17 13:22:38.281914 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:38.281894    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330fef4b-5bd2-4dc2-a56f-f73350b8b29a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "330fef4b-5bd2-4dc2-a56f-f73350b8b29a" (UID: "330fef4b-5bd2-4dc2-a56f-f73350b8b29a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:22:38.382380 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:38.382318    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/330fef4b-5bd2-4dc2-a56f-f73350b8b29a-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:22:39.045670 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:39.045646    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9"
Jul 17 13:22:39.045830 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:39.045649    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9" event={"ID":"330fef4b-5bd2-4dc2-a56f-f73350b8b29a","Type":"ContainerDied","Data":"6a8bf16592e1d1627a1c9e9b4f47f9bdd717e7b5d74982d4b09024c0e833e428"}
Jul 17 13:22:39.045830 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:39.045792    2569 scope.go:117] "RemoveContainer" containerID="8cc4e1fac787a963b7abc5ae93a8e774fce4505202d7bc707ca9d2624a394783"
Jul 17 13:22:39.068471 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:39.068445    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9"]
Jul 17 13:22:39.073565 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:39.073542    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-zfsb9"]
Jul 17 13:22:39.634266 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:39.634242    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="330fef4b-5bd2-4dc2-a56f-f73350b8b29a" path="/var/lib/kubelet/pods/330fef4b-5bd2-4dc2-a56f-f73350b8b29a/volumes"
Jul 17 13:22:42.413374 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:42.413328    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw"]
Jul 17 13:22:42.413947 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:42.413771    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="330fef4b-5bd2-4dc2-a56f-f73350b8b29a" containerName="revoke-keys"
Jul 17 13:22:42.413947 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:42.413811    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="330fef4b-5bd2-4dc2-a56f-f73350b8b29a" containerName="revoke-keys"
Jul 17 13:22:42.413947 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:42.413820    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="330fef4b-5bd2-4dc2-a56f-f73350b8b29a" containerName="revoke-keys"
Jul 17 13:22:42.413947 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:42.413826    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="330fef4b-5bd2-4dc2-a56f-f73350b8b29a" containerName="revoke-keys"
Jul 17 13:22:42.413947 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:42.413841    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="330fef4b-5bd2-4dc2-a56f-f73350b8b29a" containerName="revoke-keys"
Jul 17 13:22:42.413947 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:42.413848    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="330fef4b-5bd2-4dc2-a56f-f73350b8b29a" containerName="revoke-keys"
Jul 17 13:22:42.413947 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:42.413901    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="330fef4b-5bd2-4dc2-a56f-f73350b8b29a" containerName="revoke-keys"
Jul 17 13:22:42.413947 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:42.413908    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="330fef4b-5bd2-4dc2-a56f-f73350b8b29a" containerName="revoke-keys"
Jul 17 13:22:42.418689 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:42.418665    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw"
Jul 17 13:22:42.428882 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:42.428853    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw"]
Jul 17 13:22:42.511329 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:42.511300    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3851992-f6ab-4074-8643-b9a906644a6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw\" (UID: \"d3851992-f6ab-4074-8643-b9a906644a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw"
Jul 17 13:22:42.612089 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:42.612067    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3851992-f6ab-4074-8643-b9a906644a6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw\" (UID: \"d3851992-f6ab-4074-8643-b9a906644a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw"
Jul 17 13:22:42.612649 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:42.612629    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3851992-f6ab-4074-8643-b9a906644a6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw\" (UID: \"d3851992-f6ab-4074-8643-b9a906644a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw"
Jul 17 13:22:42.731265 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:42.731241    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw"
Jul 17 13:22:42.857625 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:42.857590    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw"]
Jul 17 13:22:42.860490 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:22:42.860464    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3851992_f6ab_4074_8643_b9a906644a6c.slice/crio-0766f70bac0eb1661da837fa3f766d8f058e12d721db88e0b4a7fda0339dea67 WatchSource:0}: Error finding container 0766f70bac0eb1661da837fa3f766d8f058e12d721db88e0b4a7fda0339dea67: Status 404 returned error can't find the container with id 0766f70bac0eb1661da837fa3f766d8f058e12d721db88e0b4a7fda0339dea67
Jul 17 13:22:43.067984 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:43.067951    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw" event={"ID":"d3851992-f6ab-4074-8643-b9a906644a6c","Type":"ContainerStarted","Data":"1955a97549675c67c666759e69f5d735fd6d67a7a9b83838ac73f270ce504227"}
Jul 17 13:22:43.068081 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:43.068010    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw" event={"ID":"d3851992-f6ab-4074-8643-b9a906644a6c","Type":"ContainerStarted","Data":"0766f70bac0eb1661da837fa3f766d8f058e12d721db88e0b4a7fda0339dea67"}
Jul 17 13:22:43.094333 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:43.094292    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw" podStartSLOduration=1.094277888 podStartE2EDuration="1.094277888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:22:43.091998689 +0000 UTC m=+3638.096045571" watchObservedRunningTime="2026-07-17 13:22:43.094277888 +0000 UTC m=+3638.098324769"
Jul 17 13:22:44.074010 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:44.073969    2569 generic.go:356] "Generic (PLEG): container finished" podID="d3851992-f6ab-4074-8643-b9a906644a6c" containerID="1955a97549675c67c666759e69f5d735fd6d67a7a9b83838ac73f270ce504227" exitCode=6
Jul 17 13:22:44.074410 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:44.074028    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw" event={"ID":"d3851992-f6ab-4074-8643-b9a906644a6c","Type":"ContainerDied","Data":"1955a97549675c67c666759e69f5d735fd6d67a7a9b83838ac73f270ce504227"}
Jul 17 13:22:44.074410 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:44.074191    2569 scope.go:117] "RemoveContainer" containerID="1955a97549675c67c666759e69f5d735fd6d67a7a9b83838ac73f270ce504227"
Jul 17 13:22:45.080624 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:45.080594    2569 generic.go:356] "Generic (PLEG): container finished" podID="d3851992-f6ab-4074-8643-b9a906644a6c" containerID="540fa5f77b0251473ba9ee310f16368b0101b47de3655f4ea6c44ec226e40902" exitCode=6
Jul 17 13:22:45.080972 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:45.080675    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw" event={"ID":"d3851992-f6ab-4074-8643-b9a906644a6c","Type":"ContainerDied","Data":"540fa5f77b0251473ba9ee310f16368b0101b47de3655f4ea6c44ec226e40902"}
Jul 17 13:22:45.080972 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:45.080761    2569 scope.go:117] "RemoveContainer" containerID="1955a97549675c67c666759e69f5d735fd6d67a7a9b83838ac73f270ce504227"
Jul 17 13:22:45.080972 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:45.080926    2569 scope.go:117] "RemoveContainer" containerID="540fa5f77b0251473ba9ee310f16368b0101b47de3655f4ea6c44ec226e40902"
Jul 17 13:22:45.081164 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:22:45.081136    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw_odh-ai-gateway-infra(d3851992-f6ab-4074-8643-b9a906644a6c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw" podUID="d3851992-f6ab-4074-8643-b9a906644a6c"
Jul 17 13:22:46.086616 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:46.086586    2569 scope.go:117] "RemoveContainer" containerID="540fa5f77b0251473ba9ee310f16368b0101b47de3655f4ea6c44ec226e40902"
Jul 17 13:22:46.087133 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:22:46.086823    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw_odh-ai-gateway-infra(d3851992-f6ab-4074-8643-b9a906644a6c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw" podUID="d3851992-f6ab-4074-8643-b9a906644a6c"
Jul 17 13:22:46.628979 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:46.628955    2569 scope.go:117] "RemoveContainer" containerID="64895264a7ff3a1276aad1ae92d2bb91cb4e26fcbd03b465b76142f46a1c829f"
Jul 17 13:22:47.093157 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:47.093125    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz" event={"ID":"a75687e5-6e3b-4363-843c-280111c2a143","Type":"ContainerStarted","Data":"d7712ca787bd8f8b033816955e1011f57655d476379587e649cd5a42d7aac5b3"}
Jul 17 13:22:47.119685 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:47.119638    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz" podStartSLOduration=17.119623751 podStartE2EDuration="17.119623751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:22:47.117236047 +0000 UTC m=+3642.121282929" watchObservedRunningTime="2026-07-17 13:22:47.119623751 +0000 UTC m=+3642.123670624"
Jul 17 13:22:48.099456 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:48.099424    2569 generic.go:356] "Generic (PLEG): container finished" podID="a75687e5-6e3b-4363-843c-280111c2a143" containerID="d7712ca787bd8f8b033816955e1011f57655d476379587e649cd5a42d7aac5b3" exitCode=6
Jul 17 13:22:48.099881 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:48.099501    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz" event={"ID":"a75687e5-6e3b-4363-843c-280111c2a143","Type":"ContainerDied","Data":"d7712ca787bd8f8b033816955e1011f57655d476379587e649cd5a42d7aac5b3"}
Jul 17 13:22:48.099881 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:48.099569    2569 scope.go:117] "RemoveContainer" containerID="64895264a7ff3a1276aad1ae92d2bb91cb4e26fcbd03b465b76142f46a1c829f"
Jul 17 13:22:48.099881 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:48.099780    2569 scope.go:117] "RemoveContainer" containerID="d7712ca787bd8f8b033816955e1011f57655d476379587e649cd5a42d7aac5b3"
Jul 17 13:22:48.100006 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:22:48.099970    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz_odh-ai-gateway-infra(a75687e5-6e3b-4363-843c-280111c2a143)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz" podUID="a75687e5-6e3b-4363-843c-280111c2a143"
Jul 17 13:22:48.132125 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:48.132099    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz"]
Jul 17 13:22:49.244548 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:49.244520    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz"
Jul 17 13:22:49.362786 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:49.362706    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75687e5-6e3b-4363-843c-280111c2a143-maas-api-service-ca\") pod \"a75687e5-6e3b-4363-843c-280111c2a143\" (UID: \"a75687e5-6e3b-4363-843c-280111c2a143\") "
Jul 17 13:22:49.363029 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:49.363007    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75687e5-6e3b-4363-843c-280111c2a143-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a75687e5-6e3b-4363-843c-280111c2a143" (UID: "a75687e5-6e3b-4363-843c-280111c2a143"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:22:49.463958 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:49.463936    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75687e5-6e3b-4363-843c-280111c2a143-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:22:50.110841 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:50.110814    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz"
Jul 17 13:22:50.111007 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:50.110809    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz" event={"ID":"a75687e5-6e3b-4363-843c-280111c2a143","Type":"ContainerDied","Data":"328fdb0e7b202c734c99f974aa132b9cf8658ee9aba1073d8c628b6634ac80ac"}
Jul 17 13:22:50.111007 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:50.110958    2569 scope.go:117] "RemoveContainer" containerID="d7712ca787bd8f8b033816955e1011f57655d476379587e649cd5a42d7aac5b3"
Jul 17 13:22:50.131064 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:50.131042    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz"]
Jul 17 13:22:50.134399 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:50.134372    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-kzxwz"]
Jul 17 13:22:51.634054 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:51.634014    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a75687e5-6e3b-4363-843c-280111c2a143" path="/var/lib/kubelet/pods/a75687e5-6e3b-4363-843c-280111c2a143/volumes"
Jul 17 13:22:52.597129 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.597100    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"]
Jul 17 13:22:52.597573 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.597559    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75687e5-6e3b-4363-843c-280111c2a143" containerName="revoke-keys"
Jul 17 13:22:52.597618 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.597576    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75687e5-6e3b-4363-843c-280111c2a143" containerName="revoke-keys"
Jul 17 13:22:52.597618 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.597612    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75687e5-6e3b-4363-843c-280111c2a143" containerName="revoke-keys"
Jul 17 13:22:52.597689 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.597622    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75687e5-6e3b-4363-843c-280111c2a143" containerName="revoke-keys"
Jul 17 13:22:52.597726 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.597697    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75687e5-6e3b-4363-843c-280111c2a143" containerName="revoke-keys"
Jul 17 13:22:52.597726 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.597707    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75687e5-6e3b-4363-843c-280111c2a143" containerName="revoke-keys"
Jul 17 13:22:52.597726 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.597719    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75687e5-6e3b-4363-843c-280111c2a143" containerName="revoke-keys"
Jul 17 13:22:52.597836 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.597729    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="330fef4b-5bd2-4dc2-a56f-f73350b8b29a" containerName="revoke-keys"
Jul 17 13:22:52.602209 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.602192    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"
Jul 17 13:22:52.605137 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.605114    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ae44b3-serving-cert\""
Jul 17 13:22:52.605295 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.605275    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lwvzj\""
Jul 17 13:22:52.606472 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.606459    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 13:22:52.612829 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.612810    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"]
Jul 17 13:22:52.687600 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.687570    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7-maas-api-tls\") pod \"maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9\" (UID: \"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"
Jul 17 13:22:52.688004 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.687610    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghzkl\" (UniqueName: \"kubernetes.io/projected/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7-kube-api-access-ghzkl\") pod \"maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9\" (UID: \"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"
Jul 17 13:22:52.688004 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.687671    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9\" (UID: \"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"
Jul 17 13:22:52.788476 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.788452    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7-maas-api-tls\") pod \"maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9\" (UID: \"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"
Jul 17 13:22:52.788586 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.788484    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghzkl\" (UniqueName: \"kubernetes.io/projected/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7-kube-api-access-ghzkl\") pod \"maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9\" (UID: \"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"
Jul 17 13:22:52.788586 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.788531    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9\" (UID: \"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"
Jul 17 13:22:52.789199 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.789178    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9\" (UID: \"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"
Jul 17 13:22:52.790669 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.790647    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7-maas-api-tls\") pod \"maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9\" (UID: \"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"
Jul 17 13:22:52.800588 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.800568    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghzkl\" (UniqueName: \"kubernetes.io/projected/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7-kube-api-access-ghzkl\") pod \"maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9\" (UID: \"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"
Jul 17 13:22:52.913254 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:52.913198    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"
Jul 17 13:22:53.248344 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:53.248322    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"]
Jul 17 13:22:53.250443 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:22:53.250418    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b1ea2f_dd14_446f_b89d_0e2a8ddfc0a7.slice/crio-cb43fe3bc552c55e90565943b33f9e6ed0eea058b4898fbe0050da933c6e5711 WatchSource:0}: Error finding container cb43fe3bc552c55e90565943b33f9e6ed0eea058b4898fbe0050da933c6e5711: Status 404 returned error can't find the container with id cb43fe3bc552c55e90565943b33f9e6ed0eea058b4898fbe0050da933c6e5711
Jul 17 13:22:54.132280 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:54.132209    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9" event={"ID":"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7","Type":"ContainerStarted","Data":"e233fee0bba84eccd5c53b1aa2e16983a866e2924f3544c0634e84880974c478"}
Jul 17 13:22:54.132280 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:54.132254    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9" event={"ID":"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7","Type":"ContainerStarted","Data":"cb43fe3bc552c55e90565943b33f9e6ed0eea058b4898fbe0050da933c6e5711"}
Jul 17 13:22:54.132637 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:54.132291    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"
Jul 17 13:22:54.153998 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:54.153955    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9" podStartSLOduration=1.631648666 podStartE2EDuration="2.153940342s" totalImagesPullingTime="522.291676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:22:54.151975899 +0000 UTC m=+3649.156022780" watchObservedRunningTime="2026-07-17 13:22:54.153940342 +0000 UTC m=+3649.157987236"
Jul 17 13:22:58.629150 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:58.629122    2569 scope.go:117] "RemoveContainer" containerID="540fa5f77b0251473ba9ee310f16368b0101b47de3655f4ea6c44ec226e40902"
Jul 17 13:22:59.157671 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:22:59.157639    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw" event={"ID":"d3851992-f6ab-4074-8643-b9a906644a6c","Type":"ContainerStarted","Data":"a79114d688101df963cf862e412ae0a871f24be573d8c5db11b6b3a97c7bcf95"}
Jul 17 13:23:00.142532 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.142499    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"
Jul 17 13:23:00.163690 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.163666    2569 generic.go:356] "Generic (PLEG): container finished" podID="d3851992-f6ab-4074-8643-b9a906644a6c" containerID="a79114d688101df963cf862e412ae0a871f24be573d8c5db11b6b3a97c7bcf95" exitCode=6
Jul 17 13:23:00.163846 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.163695    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw" event={"ID":"d3851992-f6ab-4074-8643-b9a906644a6c","Type":"ContainerDied","Data":"a79114d688101df963cf862e412ae0a871f24be573d8c5db11b6b3a97c7bcf95"}
Jul 17 13:23:00.163846 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.163732    2569 scope.go:117] "RemoveContainer" containerID="540fa5f77b0251473ba9ee310f16368b0101b47de3655f4ea6c44ec226e40902"
Jul 17 13:23:00.164020 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.164001    2569 scope.go:117] "RemoveContainer" containerID="a79114d688101df963cf862e412ae0a871f24be573d8c5db11b6b3a97c7bcf95"
Jul 17 13:23:00.164190 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:23:00.164171    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw_odh-ai-gateway-infra(d3851992-f6ab-4074-8643-b9a906644a6c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw" podUID="d3851992-f6ab-4074-8643-b9a906644a6c"
Jul 17 13:23:00.197060 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.197036    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw"]
Jul 17 13:23:00.287770 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.287724    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:23:00.623231 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.623190    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"]
Jul 17 13:23:00.623704 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.623682    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75687e5-6e3b-4363-843c-280111c2a143" containerName="revoke-keys"
Jul 17 13:23:00.623704 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.623703    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75687e5-6e3b-4363-843c-280111c2a143" containerName="revoke-keys"
Jul 17 13:23:00.627018 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.626997    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"
Jul 17 13:23:00.632562 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.632542    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5dd9aa-serving-cert\""
Jul 17 13:23:00.644193 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.644169    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"]
Jul 17 13:23:00.748710 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.748683    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-maas-api-tls\") pod \"maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv\" (UID: \"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"
Jul 17 13:23:00.748867 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.748717    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmmnw\" (UniqueName: \"kubernetes.io/projected/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-kube-api-access-kmmnw\") pod \"maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv\" (UID: \"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"
Jul 17 13:23:00.748941 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.748901    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv\" (UID: \"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"
Jul 17 13:23:00.849438 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.849410    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv\" (UID: \"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"
Jul 17 13:23:00.849552 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.849468    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-maas-api-tls\") pod \"maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv\" (UID: \"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"
Jul 17 13:23:00.849552 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.849498    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmmnw\" (UniqueName: \"kubernetes.io/projected/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-kube-api-access-kmmnw\") pod \"maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv\" (UID: \"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"
Jul 17 13:23:00.849642 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:23:00.849623    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-serving-cert: secret "maas-api-e2e-inf-b-5dd9aa-serving-cert" not found
Jul 17 13:23:00.849710 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:23:00.849698    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-maas-api-tls podName:1c70eb6d-2caa-47c8-ac41-eea1d3e0d107 nodeName:}" failed. No retries permitted until 2026-07-17 13:23:01.349676594 +0000 UTC m=+3656.353723460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-maas-api-tls") pod "maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv" (UID: "1c70eb6d-2caa-47c8-ac41-eea1d3e0d107") : secret "maas-api-e2e-inf-b-5dd9aa-serving-cert" not found
Jul 17 13:23:00.850098 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.850078    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv\" (UID: \"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"
Jul 17 13:23:00.866567 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:00.866537    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmmnw\" (UniqueName: \"kubernetes.io/projected/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-kube-api-access-kmmnw\") pod \"maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv\" (UID: \"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"
Jul 17 13:23:01.295686 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:01.295660    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw"
Jul 17 13:23:01.353220 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:01.353193    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3851992-f6ab-4074-8643-b9a906644a6c-maas-api-service-ca\") pod \"d3851992-f6ab-4074-8643-b9a906644a6c\" (UID: \"d3851992-f6ab-4074-8643-b9a906644a6c\") "
Jul 17 13:23:01.353438 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:01.353417    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-maas-api-tls\") pod \"maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv\" (UID: \"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"
Jul 17 13:23:01.353577 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:01.353535    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3851992-f6ab-4074-8643-b9a906644a6c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3851992-f6ab-4074-8643-b9a906644a6c" (UID: "d3851992-f6ab-4074-8643-b9a906644a6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:23:01.355694 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:01.355677    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-maas-api-tls\") pod \"maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv\" (UID: \"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"
Jul 17 13:23:01.454573 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:01.454520    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3851992-f6ab-4074-8643-b9a906644a6c-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:23:01.539505 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:01.539483    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"
Jul 17 13:23:01.676037 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:01.676007    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"]
Jul 17 13:23:01.678374 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:23:01.678347    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c70eb6d_2caa_47c8_ac41_eea1d3e0d107.slice/crio-430b27565ee2ac842fc6719d06643e1c2039ecb588ea0611f5b2ce11f2454698 WatchSource:0}: Error finding container 430b27565ee2ac842fc6719d06643e1c2039ecb588ea0611f5b2ce11f2454698: Status 404 returned error can't find the container with id 430b27565ee2ac842fc6719d06643e1c2039ecb588ea0611f5b2ce11f2454698
Jul 17 13:23:02.175228 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:02.175204    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw"
Jul 17 13:23:02.175228 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:02.175210    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw" event={"ID":"d3851992-f6ab-4074-8643-b9a906644a6c","Type":"ContainerDied","Data":"0766f70bac0eb1661da837fa3f766d8f058e12d721db88e0b4a7fda0339dea67"}
Jul 17 13:23:02.175410 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:02.175290    2569 scope.go:117] "RemoveContainer" containerID="a79114d688101df963cf862e412ae0a871f24be573d8c5db11b6b3a97c7bcf95"
Jul 17 13:23:02.176947 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:02.176915    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv" event={"ID":"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107","Type":"ContainerStarted","Data":"8a21a67bcf6d7444485908d6ea3c52978f6d606046614f6542d966ab86aa27d0"}
Jul 17 13:23:02.177076 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:02.176968    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv" event={"ID":"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107","Type":"ContainerStarted","Data":"430b27565ee2ac842fc6719d06643e1c2039ecb588ea0611f5b2ce11f2454698"}
Jul 17 13:23:02.177076 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:02.177030    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"
Jul 17 13:23:02.193680 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:02.193657    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw"]
Jul 17 13:23:02.196361 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:02.196342    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-p6fqw"]
Jul 17 13:23:02.215809 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:02.215766    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv" podStartSLOduration=1.834093937 podStartE2EDuration="2.215738285s" totalImagesPullingTime="381.644348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:23:02.214400944 +0000 UTC m=+3657.218447825" watchObservedRunningTime="2026-07-17 13:23:02.215738285 +0000 UTC m=+3657.219785166"
Jul 17 13:23:03.633696 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:03.633661    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3851992-f6ab-4074-8643-b9a906644a6c" path="/var/lib/kubelet/pods/d3851992-f6ab-4074-8643-b9a906644a6c/volumes"
Jul 17 13:23:08.188621 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:08.188592    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"
Jul 17 13:23:10.173131 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:10.173091    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:23:19.010857 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:19.010823    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:23:30.110385 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:30.110295    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp"]
Jul 17 13:23:30.110887 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:30.110843    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3851992-f6ab-4074-8643-b9a906644a6c" containerName="revoke-keys"
Jul 17 13:23:30.110887 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:30.110863    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3851992-f6ab-4074-8643-b9a906644a6c" containerName="revoke-keys"
Jul 17 13:23:30.110887 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:30.110887    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3851992-f6ab-4074-8643-b9a906644a6c" containerName="revoke-keys"
Jul 17 13:23:30.111070 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:30.110895    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3851992-f6ab-4074-8643-b9a906644a6c" containerName="revoke-keys"
Jul 17 13:23:30.111070 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:30.110988    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3851992-f6ab-4074-8643-b9a906644a6c" containerName="revoke-keys"
Jul 17 13:23:30.111070 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:30.111008    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3851992-f6ab-4074-8643-b9a906644a6c" containerName="revoke-keys"
Jul 17 13:23:30.114127 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:30.114102    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp"
Jul 17 13:23:30.117058 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:30.117038    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:23:30.122964 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:30.122934    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp"]
Jul 17 13:23:30.266974 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:30.266943    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b3c90f-5d08-4986-b6d1-ba191f52be1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp\" (UID: \"e3b3c90f-5d08-4986-b6d1-ba191f52be1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp"
Jul 17 13:23:30.367421 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:30.367358    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b3c90f-5d08-4986-b6d1-ba191f52be1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp\" (UID: \"e3b3c90f-5d08-4986-b6d1-ba191f52be1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp"
Jul 17 13:23:30.367938 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:30.367920    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b3c90f-5d08-4986-b6d1-ba191f52be1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp\" (UID: \"e3b3c90f-5d08-4986-b6d1-ba191f52be1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp"
Jul 17 13:23:30.426167 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:30.426142    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp"
Jul 17 13:23:30.557558 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:30.557534    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp"]
Jul 17 13:23:30.560234 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:23:30.560205    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b3c90f_5d08_4986_b6d1_ba191f52be1d.slice/crio-15efdb0b46b9ef2d1b9ffc32e548d152e644e095cb7bb397977cdd28c37e139c WatchSource:0}: Error finding container 15efdb0b46b9ef2d1b9ffc32e548d152e644e095cb7bb397977cdd28c37e139c: Status 404 returned error can't find the container with id 15efdb0b46b9ef2d1b9ffc32e548d152e644e095cb7bb397977cdd28c37e139c
Jul 17 13:23:31.321572 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:31.321546    2569 generic.go:356] "Generic (PLEG): container finished" podID="e3b3c90f-5d08-4986-b6d1-ba191f52be1d" containerID="7a041eaab9c4cf5fd2a9245c137310473859085ba35a89f66070bd2d9aa092cd" exitCode=6
Jul 17 13:23:31.321896 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:31.321597    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp" event={"ID":"e3b3c90f-5d08-4986-b6d1-ba191f52be1d","Type":"ContainerDied","Data":"7a041eaab9c4cf5fd2a9245c137310473859085ba35a89f66070bd2d9aa092cd"}
Jul 17 13:23:31.321896 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:31.321640    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp" event={"ID":"e3b3c90f-5d08-4986-b6d1-ba191f52be1d","Type":"ContainerStarted","Data":"15efdb0b46b9ef2d1b9ffc32e548d152e644e095cb7bb397977cdd28c37e139c"}
Jul 17 13:23:31.321896 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:31.321805    2569 scope.go:117] "RemoveContainer" containerID="7a041eaab9c4cf5fd2a9245c137310473859085ba35a89f66070bd2d9aa092cd"
Jul 17 13:23:32.327630 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:32.327570    2569 generic.go:356] "Generic (PLEG): container finished" podID="e3b3c90f-5d08-4986-b6d1-ba191f52be1d" containerID="3c8c39fb348a518e984df4d66136a17ee608ec60d3b432d7ef22b969272a5ab7" exitCode=6
Jul 17 13:23:32.327988 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:32.327658    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp" event={"ID":"e3b3c90f-5d08-4986-b6d1-ba191f52be1d","Type":"ContainerDied","Data":"3c8c39fb348a518e984df4d66136a17ee608ec60d3b432d7ef22b969272a5ab7"}
Jul 17 13:23:32.327988 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:32.327722    2569 scope.go:117] "RemoveContainer" containerID="7a041eaab9c4cf5fd2a9245c137310473859085ba35a89f66070bd2d9aa092cd"
Jul 17 13:23:32.327988 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:32.327913    2569 scope.go:117] "RemoveContainer" containerID="3c8c39fb348a518e984df4d66136a17ee608ec60d3b432d7ef22b969272a5ab7"
Jul 17 13:23:32.328138 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:23:32.328072    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp_odh-ai-gateway-infra(e3b3c90f-5d08-4986-b6d1-ba191f52be1d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp" podUID="e3b3c90f-5d08-4986-b6d1-ba191f52be1d"
Jul 17 13:23:32.733605 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:32.733580    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd"]
Jul 17 13:23:32.733929 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:32.733916    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3851992-f6ab-4074-8643-b9a906644a6c" containerName="revoke-keys"
Jul 17 13:23:32.733981 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:32.733930    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3851992-f6ab-4074-8643-b9a906644a6c" containerName="revoke-keys"
Jul 17 13:23:32.734026 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:32.734004    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3851992-f6ab-4074-8643-b9a906644a6c" containerName="revoke-keys"
Jul 17 13:23:32.737821 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:32.737807    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd"
Jul 17 13:23:32.760153 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:32.760133    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd"]
Jul 17 13:23:32.888499 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:32.888477    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cee5d7d-83fe-4057-9887-787e076ae8d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd\" (UID: \"5cee5d7d-83fe-4057-9887-787e076ae8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd"
Jul 17 13:23:32.989286 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:32.989232    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cee5d7d-83fe-4057-9887-787e076ae8d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd\" (UID: \"5cee5d7d-83fe-4057-9887-787e076ae8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd"
Jul 17 13:23:32.989809 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:32.989791    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cee5d7d-83fe-4057-9887-787e076ae8d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd\" (UID: \"5cee5d7d-83fe-4057-9887-787e076ae8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd"
Jul 17 13:23:33.046832 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:33.046814    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd"
Jul 17 13:23:33.333662 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:33.333595    2569 scope.go:117] "RemoveContainer" containerID="3c8c39fb348a518e984df4d66136a17ee608ec60d3b432d7ef22b969272a5ab7"
Jul 17 13:23:33.334075 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:23:33.333783    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp_odh-ai-gateway-infra(e3b3c90f-5d08-4986-b6d1-ba191f52be1d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp" podUID="e3b3c90f-5d08-4986-b6d1-ba191f52be1d"
Jul 17 13:23:33.382502 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:33.382481    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd"]
Jul 17 13:23:33.384620 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:23:33.384598    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cee5d7d_83fe_4057_9887_787e076ae8d4.slice/crio-7efae00370fb13f9366153b1ef85e61c63dd6b49737e5799159fb5642ee755da WatchSource:0}: Error finding container 7efae00370fb13f9366153b1ef85e61c63dd6b49737e5799159fb5642ee755da: Status 404 returned error can't find the container with id 7efae00370fb13f9366153b1ef85e61c63dd6b49737e5799159fb5642ee755da
Jul 17 13:23:34.341295 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:34.341217    2569 generic.go:356] "Generic (PLEG): container finished" podID="5cee5d7d-83fe-4057-9887-787e076ae8d4" containerID="12d553a7cf6840c7302b4aba37519e3453526537209556cfd7bed26467bf0c55" exitCode=6
Jul 17 13:23:34.341295 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:34.341265    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd" event={"ID":"5cee5d7d-83fe-4057-9887-787e076ae8d4","Type":"ContainerDied","Data":"12d553a7cf6840c7302b4aba37519e3453526537209556cfd7bed26467bf0c55"}
Jul 17 13:23:34.341671 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:34.341339    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd" event={"ID":"5cee5d7d-83fe-4057-9887-787e076ae8d4","Type":"ContainerStarted","Data":"7efae00370fb13f9366153b1ef85e61c63dd6b49737e5799159fb5642ee755da"}
Jul 17 13:23:34.341671 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:34.341499    2569 scope.go:117] "RemoveContainer" containerID="12d553a7cf6840c7302b4aba37519e3453526537209556cfd7bed26467bf0c55"
Jul 17 13:23:35.348626 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:35.348593    2569 generic.go:356] "Generic (PLEG): container finished" podID="5cee5d7d-83fe-4057-9887-787e076ae8d4" containerID="7623c5d8432a376172035b00c10b022f39badb20dec94b9c4bffc37e1488a808" exitCode=6
Jul 17 13:23:35.349022 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:35.348683    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd" event={"ID":"5cee5d7d-83fe-4057-9887-787e076ae8d4","Type":"ContainerDied","Data":"7623c5d8432a376172035b00c10b022f39badb20dec94b9c4bffc37e1488a808"}
Jul 17 13:23:35.349022 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:35.348763    2569 scope.go:117] "RemoveContainer" containerID="12d553a7cf6840c7302b4aba37519e3453526537209556cfd7bed26467bf0c55"
Jul 17 13:23:35.349022 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:35.348939    2569 scope.go:117] "RemoveContainer" containerID="7623c5d8432a376172035b00c10b022f39badb20dec94b9c4bffc37e1488a808"
Jul 17 13:23:35.349155 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:23:35.349135    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd_odh-ai-gateway-infra(5cee5d7d-83fe-4057-9887-787e076ae8d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd" podUID="5cee5d7d-83fe-4057-9887-787e076ae8d4"
Jul 17 13:23:36.355659 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:36.355632    2569 scope.go:117] "RemoveContainer" containerID="7623c5d8432a376172035b00c10b022f39badb20dec94b9c4bffc37e1488a808"
Jul 17 13:23:36.356105 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:23:36.355823    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd_odh-ai-gateway-infra(5cee5d7d-83fe-4057-9887-787e076ae8d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd" podUID="5cee5d7d-83fe-4057-9887-787e076ae8d4"
Jul 17 13:23:47.628947 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:47.628909    2569 scope.go:117] "RemoveContainer" containerID="3c8c39fb348a518e984df4d66136a17ee608ec60d3b432d7ef22b969272a5ab7"
Jul 17 13:23:48.415630 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:48.415604    2569 generic.go:356] "Generic (PLEG): container finished" podID="e3b3c90f-5d08-4986-b6d1-ba191f52be1d" containerID="13e293ddc5b3a2697726e7df227deccba30ea597bdab935f03f609d2fd774622" exitCode=6
Jul 17 13:23:48.415784 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:48.415693    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp" event={"ID":"e3b3c90f-5d08-4986-b6d1-ba191f52be1d","Type":"ContainerDied","Data":"13e293ddc5b3a2697726e7df227deccba30ea597bdab935f03f609d2fd774622"}
Jul 17 13:23:48.415784 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:48.415780    2569 scope.go:117] "RemoveContainer" containerID="3c8c39fb348a518e984df4d66136a17ee608ec60d3b432d7ef22b969272a5ab7"
Jul 17 13:23:48.416034 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:48.416017    2569 scope.go:117] "RemoveContainer" containerID="13e293ddc5b3a2697726e7df227deccba30ea597bdab935f03f609d2fd774622"
Jul 17 13:23:48.416253 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:23:48.416217    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp_odh-ai-gateway-infra(e3b3c90f-5d08-4986-b6d1-ba191f52be1d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp" podUID="e3b3c90f-5d08-4986-b6d1-ba191f52be1d"
Jul 17 13:23:49.445902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:49.445868    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp"]
Jul 17 13:23:49.577133 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:49.577110    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp"
Jul 17 13:23:49.629071 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:49.629046    2569 scope.go:117] "RemoveContainer" containerID="7623c5d8432a376172035b00c10b022f39badb20dec94b9c4bffc37e1488a808"
Jul 17 13:23:49.712740 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:49.712688    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b3c90f-5d08-4986-b6d1-ba191f52be1d-maas-api-service-ca\") pod \"e3b3c90f-5d08-4986-b6d1-ba191f52be1d\" (UID: \"e3b3c90f-5d08-4986-b6d1-ba191f52be1d\") "
Jul 17 13:23:49.713001 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:49.712979    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b3c90f-5d08-4986-b6d1-ba191f52be1d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e3b3c90f-5d08-4986-b6d1-ba191f52be1d" (UID: "e3b3c90f-5d08-4986-b6d1-ba191f52be1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:23:49.813736 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:49.813715    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3b3c90f-5d08-4986-b6d1-ba191f52be1d-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:23:50.427552 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:50.427527    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp"
Jul 17 13:23:50.427696 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:50.427541    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp" event={"ID":"e3b3c90f-5d08-4986-b6d1-ba191f52be1d","Type":"ContainerDied","Data":"15efdb0b46b9ef2d1b9ffc32e548d152e644e095cb7bb397977cdd28c37e139c"}
Jul 17 13:23:50.427696 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:50.427648    2569 scope.go:117] "RemoveContainer" containerID="13e293ddc5b3a2697726e7df227deccba30ea597bdab935f03f609d2fd774622"
Jul 17 13:23:50.429562 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:50.429531    2569 generic.go:356] "Generic (PLEG): container finished" podID="5cee5d7d-83fe-4057-9887-787e076ae8d4" containerID="4511b8ccfbf521905f69c7df120eabc1d0d89d7e1d174596279696f584fc7dcd" exitCode=6
Jul 17 13:23:50.429695 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:50.429556    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd" event={"ID":"5cee5d7d-83fe-4057-9887-787e076ae8d4","Type":"ContainerDied","Data":"4511b8ccfbf521905f69c7df120eabc1d0d89d7e1d174596279696f584fc7dcd"}
Jul 17 13:23:50.429836 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:50.429821    2569 scope.go:117] "RemoveContainer" containerID="4511b8ccfbf521905f69c7df120eabc1d0d89d7e1d174596279696f584fc7dcd"
Jul 17 13:23:50.430015 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:23:50.429994    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd_odh-ai-gateway-infra(5cee5d7d-83fe-4057-9887-787e076ae8d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd" podUID="5cee5d7d-83fe-4057-9887-787e076ae8d4"
Jul 17 13:23:50.435913 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:50.435896    2569 scope.go:117] "RemoveContainer" containerID="7623c5d8432a376172035b00c10b022f39badb20dec94b9c4bffc37e1488a808"
Jul 17 13:23:50.461846 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:50.461823    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp"]
Jul 17 13:23:50.467629 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:50.467611    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8lcxp"]
Jul 17 13:23:51.462256 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:51.462220    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd"]
Jul 17 13:23:51.596523 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:51.596498    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd"
Jul 17 13:23:51.627788 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:51.627732    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cee5d7d-83fe-4057-9887-787e076ae8d4-maas-api-service-ca\") pod \"5cee5d7d-83fe-4057-9887-787e076ae8d4\" (UID: \"5cee5d7d-83fe-4057-9887-787e076ae8d4\") "
Jul 17 13:23:51.628015 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:51.627995    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cee5d7d-83fe-4057-9887-787e076ae8d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5cee5d7d-83fe-4057-9887-787e076ae8d4" (UID: "5cee5d7d-83fe-4057-9887-787e076ae8d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:23:51.628109 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:51.628089    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cee5d7d-83fe-4057-9887-787e076ae8d4-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:23:51.633989 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:51.633963    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3b3c90f-5d08-4986-b6d1-ba191f52be1d" path="/var/lib/kubelet/pods/e3b3c90f-5d08-4986-b6d1-ba191f52be1d/volumes"
Jul 17 13:23:52.444553 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:52.444526    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd"
Jul 17 13:23:52.444697 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:52.444567    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd" event={"ID":"5cee5d7d-83fe-4057-9887-787e076ae8d4","Type":"ContainerDied","Data":"7efae00370fb13f9366153b1ef85e61c63dd6b49737e5799159fb5642ee755da"}
Jul 17 13:23:52.444697 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:52.444639    2569 scope.go:117] "RemoveContainer" containerID="4511b8ccfbf521905f69c7df120eabc1d0d89d7e1d174596279696f584fc7dcd"
Jul 17 13:23:52.464069 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:52.464041    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd"]
Jul 17 13:23:52.469288 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:52.469264    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-cfffd"]
Jul 17 13:23:53.633672 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:23:53.633643    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cee5d7d-83fe-4057-9887-787e076ae8d4" path="/var/lib/kubelet/pods/5cee5d7d-83fe-4057-9887-787e076ae8d4/volumes"
Jul 17 13:24:19.475341 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:19.475292    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:24:22.809046 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.809004    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9"]
Jul 17 13:24:22.809546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.809370    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cee5d7d-83fe-4057-9887-787e076ae8d4" containerName="revoke-keys"
Jul 17 13:24:22.809546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.809383    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cee5d7d-83fe-4057-9887-787e076ae8d4" containerName="revoke-keys"
Jul 17 13:24:22.809546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.809408    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b3c90f-5d08-4986-b6d1-ba191f52be1d" containerName="revoke-keys"
Jul 17 13:24:22.809546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.809414    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b3c90f-5d08-4986-b6d1-ba191f52be1d" containerName="revoke-keys"
Jul 17 13:24:22.809546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.809420    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b3c90f-5d08-4986-b6d1-ba191f52be1d" containerName="revoke-keys"
Jul 17 13:24:22.809546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.809426    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b3c90f-5d08-4986-b6d1-ba191f52be1d" containerName="revoke-keys"
Jul 17 13:24:22.809546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.809433    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cee5d7d-83fe-4057-9887-787e076ae8d4" containerName="revoke-keys"
Jul 17 13:24:22.809546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.809438    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cee5d7d-83fe-4057-9887-787e076ae8d4" containerName="revoke-keys"
Jul 17 13:24:22.809546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.809447    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b3c90f-5d08-4986-b6d1-ba191f52be1d" containerName="revoke-keys"
Jul 17 13:24:22.809546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.809452    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b3c90f-5d08-4986-b6d1-ba191f52be1d" containerName="revoke-keys"
Jul 17 13:24:22.809546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.809505    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cee5d7d-83fe-4057-9887-787e076ae8d4" containerName="revoke-keys"
Jul 17 13:24:22.809546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.809513    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b3c90f-5d08-4986-b6d1-ba191f52be1d" containerName="revoke-keys"
Jul 17 13:24:22.809546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.809519    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b3c90f-5d08-4986-b6d1-ba191f52be1d" containerName="revoke-keys"
Jul 17 13:24:22.809546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.809526    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b3c90f-5d08-4986-b6d1-ba191f52be1d" containerName="revoke-keys"
Jul 17 13:24:22.809546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.809533    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cee5d7d-83fe-4057-9887-787e076ae8d4" containerName="revoke-keys"
Jul 17 13:24:22.812602 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.812580    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9"
Jul 17 13:24:22.816945 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.816921    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:24:22.834881 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.834854    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9"]
Jul 17 13:24:22.852435 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.852411    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f972d0c8-5c4c-4eb2-a1ab-6e270199ab17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9\" (UID: \"f972d0c8-5c4c-4eb2-a1ab-6e270199ab17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9"
Jul 17 13:24:22.953770 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.953723    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f972d0c8-5c4c-4eb2-a1ab-6e270199ab17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9\" (UID: \"f972d0c8-5c4c-4eb2-a1ab-6e270199ab17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9"
Jul 17 13:24:22.954305 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:22.954283    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f972d0c8-5c4c-4eb2-a1ab-6e270199ab17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9\" (UID: \"f972d0c8-5c4c-4eb2-a1ab-6e270199ab17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9"
Jul 17 13:24:23.123233 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:23.123173    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9"
Jul 17 13:24:23.259837 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:23.259814    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9"]
Jul 17 13:24:23.261423 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:24:23.261398    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf972d0c8_5c4c_4eb2_a1ab_6e270199ab17.slice/crio-7014f81c694ffc4ac30569c15c32f4e7e172bfa187f19d43c64e2e39a9b96c26 WatchSource:0}: Error finding container 7014f81c694ffc4ac30569c15c32f4e7e172bfa187f19d43c64e2e39a9b96c26: Status 404 returned error can't find the container with id 7014f81c694ffc4ac30569c15c32f4e7e172bfa187f19d43c64e2e39a9b96c26
Jul 17 13:24:23.596274 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:23.596241    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9" event={"ID":"f972d0c8-5c4c-4eb2-a1ab-6e270199ab17","Type":"ContainerStarted","Data":"c102b7eb88148babfbd20fabb517642db9baad69a83aaeaf3cc113a05ded2a30"}
Jul 17 13:24:23.596403 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:23.596295    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9" event={"ID":"f972d0c8-5c4c-4eb2-a1ab-6e270199ab17","Type":"ContainerStarted","Data":"7014f81c694ffc4ac30569c15c32f4e7e172bfa187f19d43c64e2e39a9b96c26"}
Jul 17 13:24:23.622524 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:23.622486    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9" podStartSLOduration=1.622472642 podStartE2EDuration="1.622472642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:24:23.619439137 +0000 UTC m=+3738.623486018" watchObservedRunningTime="2026-07-17 13:24:23.622472642 +0000 UTC m=+3738.626519524"
Jul 17 13:24:24.602722 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:24.602691    2569 generic.go:356] "Generic (PLEG): container finished" podID="f972d0c8-5c4c-4eb2-a1ab-6e270199ab17" containerID="c102b7eb88148babfbd20fabb517642db9baad69a83aaeaf3cc113a05ded2a30" exitCode=6
Jul 17 13:24:24.603117 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:24.602781    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9" event={"ID":"f972d0c8-5c4c-4eb2-a1ab-6e270199ab17","Type":"ContainerDied","Data":"c102b7eb88148babfbd20fabb517642db9baad69a83aaeaf3cc113a05ded2a30"}
Jul 17 13:24:24.603117 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:24.602918    2569 scope.go:117] "RemoveContainer" containerID="c102b7eb88148babfbd20fabb517642db9baad69a83aaeaf3cc113a05ded2a30"
Jul 17 13:24:25.609553 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:25.609521    2569 generic.go:356] "Generic (PLEG): container finished" podID="f972d0c8-5c4c-4eb2-a1ab-6e270199ab17" containerID="c5867d55f9d5474e291b64674eeb6e67c80f518c46ea9e754d20039803d7e3bc" exitCode=6
Jul 17 13:24:25.609998 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:25.609589    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9" event={"ID":"f972d0c8-5c4c-4eb2-a1ab-6e270199ab17","Type":"ContainerDied","Data":"c5867d55f9d5474e291b64674eeb6e67c80f518c46ea9e754d20039803d7e3bc"}
Jul 17 13:24:25.609998 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:25.609674    2569 scope.go:117] "RemoveContainer" containerID="c102b7eb88148babfbd20fabb517642db9baad69a83aaeaf3cc113a05ded2a30"
Jul 17 13:24:25.609998 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:25.609878    2569 scope.go:117] "RemoveContainer" containerID="c5867d55f9d5474e291b64674eeb6e67c80f518c46ea9e754d20039803d7e3bc"
Jul 17 13:24:25.610183 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:24:25.610091    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9_odh-ai-gateway-infra(f972d0c8-5c4c-4eb2-a1ab-6e270199ab17)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9" podUID="f972d0c8-5c4c-4eb2-a1ab-6e270199ab17"
Jul 17 13:24:26.616477 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:26.616447    2569 scope.go:117] "RemoveContainer" containerID="c5867d55f9d5474e291b64674eeb6e67c80f518c46ea9e754d20039803d7e3bc"
Jul 17 13:24:26.616870 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:24:26.616636    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9_odh-ai-gateway-infra(f972d0c8-5c4c-4eb2-a1ab-6e270199ab17)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9" podUID="f972d0c8-5c4c-4eb2-a1ab-6e270199ab17"
Jul 17 13:24:33.565991 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:33.565953    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:24:41.386137 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:41.386100    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2"]
Jul 17 13:24:41.386484 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:41.386423    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cee5d7d-83fe-4057-9887-787e076ae8d4" containerName="revoke-keys"
Jul 17 13:24:41.386484 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:41.386434    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cee5d7d-83fe-4057-9887-787e076ae8d4" containerName="revoke-keys"
Jul 17 13:24:41.386573 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:41.386528    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cee5d7d-83fe-4057-9887-787e076ae8d4" containerName="revoke-keys"
Jul 17 13:24:41.390607 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:41.390587    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2"
Jul 17 13:24:41.399017 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:41.398988    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2"]
Jul 17 13:24:41.491023 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:41.490991    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2\" (UID: \"647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2"
Jul 17 13:24:41.592052 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:41.592021    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2\" (UID: \"647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2"
Jul 17 13:24:41.592605 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:41.592583    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2\" (UID: \"647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2"
Jul 17 13:24:41.628307 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:41.628288    2569 scope.go:117] "RemoveContainer" containerID="c5867d55f9d5474e291b64674eeb6e67c80f518c46ea9e754d20039803d7e3bc"
Jul 17 13:24:41.701745 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:41.701727    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2"
Jul 17 13:24:41.827295 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:41.827272    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2"]
Jul 17 13:24:41.836896 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:24:41.836869    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647b3b31_f2d6_408c_ab01_c2e6cf7d0dfe.slice/crio-ca6128bc80759483712e66242c0e8080e179dd71aae7932d7547ace898698ec3 WatchSource:0}: Error finding container ca6128bc80759483712e66242c0e8080e179dd71aae7932d7547ace898698ec3: Status 404 returned error can't find the container with id ca6128bc80759483712e66242c0e8080e179dd71aae7932d7547ace898698ec3
Jul 17 13:24:42.702273 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:42.702218    2569 generic.go:356] "Generic (PLEG): container finished" podID="f972d0c8-5c4c-4eb2-a1ab-6e270199ab17" containerID="fbf8a2a64d29994c33b35ce142a7209766a0ccb74d8d38c70bf1a4de74c50e7c" exitCode=6
Jul 17 13:24:42.702622 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:42.702298    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9" event={"ID":"f972d0c8-5c4c-4eb2-a1ab-6e270199ab17","Type":"ContainerDied","Data":"fbf8a2a64d29994c33b35ce142a7209766a0ccb74d8d38c70bf1a4de74c50e7c"}
Jul 17 13:24:42.702622 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:42.702369    2569 scope.go:117] "RemoveContainer" containerID="c5867d55f9d5474e291b64674eeb6e67c80f518c46ea9e754d20039803d7e3bc"
Jul 17 13:24:42.702622 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:42.702611    2569 scope.go:117] "RemoveContainer" containerID="fbf8a2a64d29994c33b35ce142a7209766a0ccb74d8d38c70bf1a4de74c50e7c"
Jul 17 13:24:42.702830 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:24:42.702806    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9_odh-ai-gateway-infra(f972d0c8-5c4c-4eb2-a1ab-6e270199ab17)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9" podUID="f972d0c8-5c4c-4eb2-a1ab-6e270199ab17"
Jul 17 13:24:42.703947 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:42.703918    2569 generic.go:356] "Generic (PLEG): container finished" podID="647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe" containerID="3c8246fa29872744fb5866a5a4827b0f75fa863bdb47cc52fbcf5be4b5ca1050" exitCode=6
Jul 17 13:24:42.704062 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:42.703960    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2" event={"ID":"647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe","Type":"ContainerDied","Data":"3c8246fa29872744fb5866a5a4827b0f75fa863bdb47cc52fbcf5be4b5ca1050"}
Jul 17 13:24:42.704062 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:42.703992    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2" event={"ID":"647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe","Type":"ContainerStarted","Data":"ca6128bc80759483712e66242c0e8080e179dd71aae7932d7547ace898698ec3"}
Jul 17 13:24:42.704152 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:42.704140    2569 scope.go:117] "RemoveContainer" containerID="3c8246fa29872744fb5866a5a4827b0f75fa863bdb47cc52fbcf5be4b5ca1050"
Jul 17 13:24:43.709849 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:43.709773    2569 generic.go:356] "Generic (PLEG): container finished" podID="647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe" containerID="1ad18e9250e898a281691efcc03f53c4176b7d3061a992fbebc19ff4c4b7193f" exitCode=6
Jul 17 13:24:43.710220 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:43.709860    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2" event={"ID":"647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe","Type":"ContainerDied","Data":"1ad18e9250e898a281691efcc03f53c4176b7d3061a992fbebc19ff4c4b7193f"}
Jul 17 13:24:43.710220 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:43.709917    2569 scope.go:117] "RemoveContainer" containerID="3c8246fa29872744fb5866a5a4827b0f75fa863bdb47cc52fbcf5be4b5ca1050"
Jul 17 13:24:43.710220 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:43.710034    2569 scope.go:117] "RemoveContainer" containerID="1ad18e9250e898a281691efcc03f53c4176b7d3061a992fbebc19ff4c4b7193f"
Jul 17 13:24:43.710369 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:24:43.710228    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2_odh-ai-gateway-infra(647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2" podUID="647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe"
Jul 17 13:24:43.731773 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:43.731736    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9"]
Jul 17 13:24:43.864981 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:43.864961    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9"
Jul 17 13:24:44.011019 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:44.010993    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f972d0c8-5c4c-4eb2-a1ab-6e270199ab17-maas-api-service-ca\") pod \"f972d0c8-5c4c-4eb2-a1ab-6e270199ab17\" (UID: \"f972d0c8-5c4c-4eb2-a1ab-6e270199ab17\") "
Jul 17 13:24:44.011339 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:44.011316    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f972d0c8-5c4c-4eb2-a1ab-6e270199ab17-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f972d0c8-5c4c-4eb2-a1ab-6e270199ab17" (UID: "f972d0c8-5c4c-4eb2-a1ab-6e270199ab17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:24:44.111411 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:44.111379    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f972d0c8-5c4c-4eb2-a1ab-6e270199ab17-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:24:44.718248 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:44.718226    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9"
Jul 17 13:24:44.718670 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:44.718272    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9" event={"ID":"f972d0c8-5c4c-4eb2-a1ab-6e270199ab17","Type":"ContainerDied","Data":"7014f81c694ffc4ac30569c15c32f4e7e172bfa187f19d43c64e2e39a9b96c26"}
Jul 17 13:24:44.718670 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:44.718329    2569 scope.go:117] "RemoveContainer" containerID="fbf8a2a64d29994c33b35ce142a7209766a0ccb74d8d38c70bf1a4de74c50e7c"
Jul 17 13:24:44.720131 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:44.720109    2569 scope.go:117] "RemoveContainer" containerID="1ad18e9250e898a281691efcc03f53c4176b7d3061a992fbebc19ff4c4b7193f"
Jul 17 13:24:44.720277 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:24:44.720260    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2_odh-ai-gateway-infra(647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2" podUID="647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe"
Jul 17 13:24:44.758063 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:44.758039    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9"]
Jul 17 13:24:44.766963 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:44.766944    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qrkp9"]
Jul 17 13:24:45.634272 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:45.634242    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f972d0c8-5c4c-4eb2-a1ab-6e270199ab17" path="/var/lib/kubelet/pods/f972d0c8-5c4c-4eb2-a1ab-6e270199ab17/volumes"
Jul 17 13:24:47.090831 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:47.090799    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:24:51.254486 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.254444    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"]
Jul 17 13:24:51.255040 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.254694    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9" podUID="b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7" containerName="maas-api" containerID="cri-o://e233fee0bba84eccd5c53b1aa2e16983a866e2924f3544c0634e84880974c478" gracePeriod=30
Jul 17 13:24:51.501303 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.501281    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"
Jul 17 13:24:51.568078 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.568007    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghzkl\" (UniqueName: \"kubernetes.io/projected/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7-kube-api-access-ghzkl\") pod \"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7\" (UID: \"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7\") "
Jul 17 13:24:51.568078 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.568046    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7-openshift-service-ca\") pod \"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7\" (UID: \"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7\") "
Jul 17 13:24:51.568262 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.568112    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7-maas-api-tls\") pod \"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7\" (UID: \"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7\") "
Jul 17 13:24:51.568456 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.568429    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7" (UID: "b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:24:51.569935 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.569918    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7-kube-api-access-ghzkl" (OuterVolumeSpecName: "kube-api-access-ghzkl") pod "b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7" (UID: "b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7"). InnerVolumeSpecName "kube-api-access-ghzkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:24:51.570211 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.570196    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7" (UID: "b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:24:51.668914 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.668890    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghzkl\" (UniqueName: \"kubernetes.io/projected/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7-kube-api-access-ghzkl\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:24:51.668914 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.668911    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7-openshift-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:24:51.669039 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.668920    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7-maas-api-tls\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:24:51.752821 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.752790    2569 generic.go:356] "Generic (PLEG): container finished" podID="b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7" containerID="e233fee0bba84eccd5c53b1aa2e16983a866e2924f3544c0634e84880974c478" exitCode=0
Jul 17 13:24:51.752914 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.752848    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"
Jul 17 13:24:51.752914 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.752875    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9" event={"ID":"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7","Type":"ContainerDied","Data":"e233fee0bba84eccd5c53b1aa2e16983a866e2924f3544c0634e84880974c478"}
Jul 17 13:24:51.753015 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.752930    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9" event={"ID":"b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7","Type":"ContainerDied","Data":"cb43fe3bc552c55e90565943b33f9e6ed0eea058b4898fbe0050da933c6e5711"}
Jul 17 13:24:51.753015 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.752952    2569 scope.go:117] "RemoveContainer" containerID="e233fee0bba84eccd5c53b1aa2e16983a866e2924f3544c0634e84880974c478"
Jul 17 13:24:51.761348 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.761326    2569 scope.go:117] "RemoveContainer" containerID="e233fee0bba84eccd5c53b1aa2e16983a866e2924f3544c0634e84880974c478"
Jul 17 13:24:51.761622 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:24:51.761604    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e233fee0bba84eccd5c53b1aa2e16983a866e2924f3544c0634e84880974c478\": container with ID starting with e233fee0bba84eccd5c53b1aa2e16983a866e2924f3544c0634e84880974c478 not found: ID does not exist" containerID="e233fee0bba84eccd5c53b1aa2e16983a866e2924f3544c0634e84880974c478"
Jul 17 13:24:51.761680 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.761628    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e233fee0bba84eccd5c53b1aa2e16983a866e2924f3544c0634e84880974c478"} err="failed to get container status \"e233fee0bba84eccd5c53b1aa2e16983a866e2924f3544c0634e84880974c478\": rpc error: code = NotFound desc = could not find container \"e233fee0bba84eccd5c53b1aa2e16983a866e2924f3544c0634e84880974c478\": container with ID starting with e233fee0bba84eccd5c53b1aa2e16983a866e2924f3544c0634e84880974c478 not found: ID does not exist"
Jul 17 13:24:51.770434 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.770411    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"]
Jul 17 13:24:51.774845 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:51.774816    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ae44b3-b98fcb5bd-6qwc9"]
Jul 17 13:24:53.634389 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:53.634359    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7" path="/var/lib/kubelet/pods/b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7/volumes"
Jul 17 13:24:55.632407 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:55.632378    2569 scope.go:117] "RemoveContainer" containerID="1ad18e9250e898a281691efcc03f53c4176b7d3061a992fbebc19ff4c4b7193f"
Jul 17 13:24:56.780422 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:56.780395    2569 generic.go:356] "Generic (PLEG): container finished" podID="647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe" containerID="9639b3b3c0596d5e163719a44abe1d685f1e6c7f33ef23fd7cc21f158a5be96e" exitCode=6
Jul 17 13:24:56.780800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:56.780468    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2" event={"ID":"647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe","Type":"ContainerDied","Data":"9639b3b3c0596d5e163719a44abe1d685f1e6c7f33ef23fd7cc21f158a5be96e"}
Jul 17 13:24:56.780800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:56.780532    2569 scope.go:117] "RemoveContainer" containerID="1ad18e9250e898a281691efcc03f53c4176b7d3061a992fbebc19ff4c4b7193f"
Jul 17 13:24:56.780800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:56.780788    2569 scope.go:117] "RemoveContainer" containerID="9639b3b3c0596d5e163719a44abe1d685f1e6c7f33ef23fd7cc21f158a5be96e"
Jul 17 13:24:56.780996 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:24:56.780977    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2_odh-ai-gateway-infra(647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2" podUID="647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe"
Jul 17 13:24:57.812800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:57.812699    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2"]
Jul 17 13:24:57.945650 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:57.945630    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2"
Jul 17 13:24:58.015313 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:58.015285    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe-maas-api-service-ca\") pod \"647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe\" (UID: \"647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe\") "
Jul 17 13:24:58.015603 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:58.015581    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe" (UID: "647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:24:58.116666 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:58.116611    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:24:58.794086 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:58.794061    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2"
Jul 17 13:24:58.794242 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:58.794105    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2" event={"ID":"647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe","Type":"ContainerDied","Data":"ca6128bc80759483712e66242c0e8080e179dd71aae7932d7547ace898698ec3"}
Jul 17 13:24:58.794242 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:58.794166    2569 scope.go:117] "RemoveContainer" containerID="9639b3b3c0596d5e163719a44abe1d685f1e6c7f33ef23fd7cc21f158a5be96e"
Jul 17 13:24:58.822819 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:58.822791    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2"]
Jul 17 13:24:58.831292 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:58.831270    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-sz9z2"]
Jul 17 13:24:59.634287 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:24:59.634251    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe" path="/var/lib/kubelet/pods/647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe/volumes"
Jul 17 13:25:00.908285 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:00.908248    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:25:06.828814 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.828781    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm"]
Jul 17 13:25:06.829180 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.829142    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe" containerName="revoke-keys"
Jul 17 13:25:06.829180 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.829153    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe" containerName="revoke-keys"
Jul 17 13:25:06.829180 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.829164    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f972d0c8-5c4c-4eb2-a1ab-6e270199ab17" containerName="revoke-keys"
Jul 17 13:25:06.829180 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.829169    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f972d0c8-5c4c-4eb2-a1ab-6e270199ab17" containerName="revoke-keys"
Jul 17 13:25:06.829180 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.829182    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f972d0c8-5c4c-4eb2-a1ab-6e270199ab17" containerName="revoke-keys"
Jul 17 13:25:06.829347 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.829187    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f972d0c8-5c4c-4eb2-a1ab-6e270199ab17" containerName="revoke-keys"
Jul 17 13:25:06.829347 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.829198    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7" containerName="maas-api"
Jul 17 13:25:06.829347 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.829203    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7" containerName="maas-api"
Jul 17 13:25:06.829347 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.829211    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe" containerName="revoke-keys"
Jul 17 13:25:06.829347 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.829216    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe" containerName="revoke-keys"
Jul 17 13:25:06.829347 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.829267    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe" containerName="revoke-keys"
Jul 17 13:25:06.829347 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.829275    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe" containerName="revoke-keys"
Jul 17 13:25:06.829347 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.829281    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f972d0c8-5c4c-4eb2-a1ab-6e270199ab17" containerName="revoke-keys"
Jul 17 13:25:06.829347 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.829287    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f972d0c8-5c4c-4eb2-a1ab-6e270199ab17" containerName="revoke-keys"
Jul 17 13:25:06.829347 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.829292    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f972d0c8-5c4c-4eb2-a1ab-6e270199ab17" containerName="revoke-keys"
Jul 17 13:25:06.829347 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.829298    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1b1ea2f-dd14-446f-b89d-0e2a8ddfc0a7" containerName="maas-api"
Jul 17 13:25:06.833464 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.833444    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm"
Jul 17 13:25:06.837733 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.837711    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:25:06.843737 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.843711    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm"]
Jul 17 13:25:06.982590 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:06.982550    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm\" (UID: \"b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm"
Jul 17 13:25:07.083806 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:07.083699    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm\" (UID: \"b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm"
Jul 17 13:25:07.084285 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:07.084267    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm\" (UID: \"b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm"
Jul 17 13:25:07.144563 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:07.144540    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm"
Jul 17 13:25:07.485093 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:07.485013    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm"]
Jul 17 13:25:07.487865 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:25:07.487823    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f65aee_9ffc_4c0d_a44e_2f9d7a13f502.slice/crio-ff1e8969c56e7f9536b5e75cc73eb56fcfe5bffe3567fce6cf347047f837cd8c WatchSource:0}: Error finding container ff1e8969c56e7f9536b5e75cc73eb56fcfe5bffe3567fce6cf347047f837cd8c: Status 404 returned error can't find the container with id ff1e8969c56e7f9536b5e75cc73eb56fcfe5bffe3567fce6cf347047f837cd8c
Jul 17 13:25:07.845544 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:07.845518    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm" event={"ID":"b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502","Type":"ContainerStarted","Data":"410c38345de24c9dd043941d51d78b12ecc32e466c3599619109561137219338"}
Jul 17 13:25:07.845941 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:07.845563    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm" event={"ID":"b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502","Type":"ContainerStarted","Data":"ff1e8969c56e7f9536b5e75cc73eb56fcfe5bffe3567fce6cf347047f837cd8c"}
Jul 17 13:25:07.860554 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:07.860503    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm" podStartSLOduration=1.860490306 podStartE2EDuration="1.860490306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:25:07.860337046 +0000 UTC m=+3782.864383930" watchObservedRunningTime="2026-07-17 13:25:07.860490306 +0000 UTC m=+3782.864537187"
Jul 17 13:25:08.852152 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:08.852116    2569 generic.go:356] "Generic (PLEG): container finished" podID="b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502" containerID="410c38345de24c9dd043941d51d78b12ecc32e466c3599619109561137219338" exitCode=0
Jul 17 13:25:08.852528 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:08.852193    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm" event={"ID":"b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502","Type":"ContainerDied","Data":"410c38345de24c9dd043941d51d78b12ecc32e466c3599619109561137219338"}
Jul 17 13:25:09.988121 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:09.988100    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm"
Jul 17 13:25:10.104568 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:10.104537    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502-maas-api-service-ca\") pod \"b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502\" (UID: \"b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502\") "
Jul 17 13:25:10.104900 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:10.104878    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502" (UID: "b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:25:10.205892 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:10.205826    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:25:10.862418 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:10.862390    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm"
Jul 17 13:25:10.862553 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:10.862387    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm" event={"ID":"b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502","Type":"ContainerDied","Data":"ff1e8969c56e7f9536b5e75cc73eb56fcfe5bffe3567fce6cf347047f837cd8c"}
Jul 17 13:25:10.862553 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:10.862515    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff1e8969c56e7f9536b5e75cc73eb56fcfe5bffe3567fce6cf347047f837cd8c"
Jul 17 13:25:17.261622 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.261583    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"]
Jul 17 13:25:17.262575 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.262505    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv" podUID="1c70eb6d-2caa-47c8-ac41-eea1d3e0d107" containerName="maas-api" containerID="cri-o://8a21a67bcf6d7444485908d6ea3c52978f6d606046614f6542d966ab86aa27d0" gracePeriod=30
Jul 17 13:25:17.527505 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.527481    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"
Jul 17 13:25:17.663017 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.662994    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-maas-api-tls\") pod \"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107\" (UID: \"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107\") "
Jul 17 13:25:17.663145 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.663028    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-openshift-service-ca\") pod \"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107\" (UID: \"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107\") "
Jul 17 13:25:17.663145 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.663080    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmmnw\" (UniqueName: \"kubernetes.io/projected/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-kube-api-access-kmmnw\") pod \"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107\" (UID: \"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107\") "
Jul 17 13:25:17.663429 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.663406    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1c70eb6d-2caa-47c8-ac41-eea1d3e0d107" (UID: "1c70eb6d-2caa-47c8-ac41-eea1d3e0d107"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:25:17.665006 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.664981    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1c70eb6d-2caa-47c8-ac41-eea1d3e0d107" (UID: "1c70eb6d-2caa-47c8-ac41-eea1d3e0d107"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:25:17.665074 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.665019    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-kube-api-access-kmmnw" (OuterVolumeSpecName: "kube-api-access-kmmnw") pod "1c70eb6d-2caa-47c8-ac41-eea1d3e0d107" (UID: "1c70eb6d-2caa-47c8-ac41-eea1d3e0d107"). InnerVolumeSpecName "kube-api-access-kmmnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:25:17.764605 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.764550    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-maas-api-tls\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:25:17.764605 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.764575    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-openshift-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:25:17.764605 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.764589    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmmnw\" (UniqueName: \"kubernetes.io/projected/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107-kube-api-access-kmmnw\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:25:17.899574 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.899547    2569 generic.go:356] "Generic (PLEG): container finished" podID="1c70eb6d-2caa-47c8-ac41-eea1d3e0d107" containerID="8a21a67bcf6d7444485908d6ea3c52978f6d606046614f6542d966ab86aa27d0" exitCode=0
Jul 17 13:25:17.899685 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.899580    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv" event={"ID":"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107","Type":"ContainerDied","Data":"8a21a67bcf6d7444485908d6ea3c52978f6d606046614f6542d966ab86aa27d0"}
Jul 17 13:25:17.899685 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.899590    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"
Jul 17 13:25:17.899685 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.899633    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv" event={"ID":"1c70eb6d-2caa-47c8-ac41-eea1d3e0d107","Type":"ContainerDied","Data":"430b27565ee2ac842fc6719d06643e1c2039ecb588ea0611f5b2ce11f2454698"}
Jul 17 13:25:17.899685 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.899665    2569 scope.go:117] "RemoveContainer" containerID="8a21a67bcf6d7444485908d6ea3c52978f6d606046614f6542d966ab86aa27d0"
Jul 17 13:25:17.908220 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.908203    2569 scope.go:117] "RemoveContainer" containerID="8a21a67bcf6d7444485908d6ea3c52978f6d606046614f6542d966ab86aa27d0"
Jul 17 13:25:17.908483 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:25:17.908461    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a21a67bcf6d7444485908d6ea3c52978f6d606046614f6542d966ab86aa27d0\": container with ID starting with 8a21a67bcf6d7444485908d6ea3c52978f6d606046614f6542d966ab86aa27d0 not found: ID does not exist" containerID="8a21a67bcf6d7444485908d6ea3c52978f6d606046614f6542d966ab86aa27d0"
Jul 17 13:25:17.908565 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.908491    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a21a67bcf6d7444485908d6ea3c52978f6d606046614f6542d966ab86aa27d0"} err="failed to get container status \"8a21a67bcf6d7444485908d6ea3c52978f6d606046614f6542d966ab86aa27d0\": rpc error: code = NotFound desc = could not find container \"8a21a67bcf6d7444485908d6ea3c52978f6d606046614f6542d966ab86aa27d0\": container with ID starting with 8a21a67bcf6d7444485908d6ea3c52978f6d606046614f6542d966ab86aa27d0 not found: ID does not exist"
Jul 17 13:25:17.921375 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.921347    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"]
Jul 17 13:25:17.925264 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:17.925243    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5dd9aa-6d544f9598-5z7hv"]
Jul 17 13:25:18.903631 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:18.903600    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt"]
Jul 17 13:25:18.904000 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:18.903935    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f972d0c8-5c4c-4eb2-a1ab-6e270199ab17" containerName="revoke-keys"
Jul 17 13:25:18.904000 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:18.903947    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f972d0c8-5c4c-4eb2-a1ab-6e270199ab17" containerName="revoke-keys"
Jul 17 13:25:18.904000 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:18.903968    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe" containerName="revoke-keys"
Jul 17 13:25:18.904000 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:18.903974    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe" containerName="revoke-keys"
Jul 17 13:25:18.904000 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:18.903984    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c70eb6d-2caa-47c8-ac41-eea1d3e0d107" containerName="maas-api"
Jul 17 13:25:18.904000 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:18.903990    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c70eb6d-2caa-47c8-ac41-eea1d3e0d107" containerName="maas-api"
Jul 17 13:25:18.904000 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:18.903999    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502" containerName="revoke-keys"
Jul 17 13:25:18.904000 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:18.904005    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502" containerName="revoke-keys"
Jul 17 13:25:18.904252 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:18.904056    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c70eb6d-2caa-47c8-ac41-eea1d3e0d107" containerName="maas-api"
Jul 17 13:25:18.904252 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:18.904065    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502" containerName="revoke-keys"
Jul 17 13:25:18.904252 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:18.904072    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="647b3b31-f2d6-408c-ab01-c2e6cf7d0dfe" containerName="revoke-keys"
Jul 17 13:25:18.908452 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:18.908431    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt"
Jul 17 13:25:18.911681 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:18.911632    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:25:18.913106 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:18.912907    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:25:18.915246 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:18.915225    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt"]
Jul 17 13:25:19.075028 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:19.074998    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f111151d-c379-4472-8c58-505e60313943-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt\" (UID: \"f111151d-c379-4472-8c58-505e60313943\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt"
Jul 17 13:25:19.176075 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:19.176015    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f111151d-c379-4472-8c58-505e60313943-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt\" (UID: \"f111151d-c379-4472-8c58-505e60313943\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt"
Jul 17 13:25:19.176598 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:19.176578    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f111151d-c379-4472-8c58-505e60313943-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt\" (UID: \"f111151d-c379-4472-8c58-505e60313943\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt"
Jul 17 13:25:19.220693 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:19.220671    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt"
Jul 17 13:25:19.349521 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:19.349499    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt"]
Jul 17 13:25:19.351468 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:25:19.351441    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf111151d_c379_4472_8c58_505e60313943.slice/crio-a348a602a643eaaa8acbe074517844b5a0d0e98d33be86fc4f5073fd42ed5e9c WatchSource:0}: Error finding container a348a602a643eaaa8acbe074517844b5a0d0e98d33be86fc4f5073fd42ed5e9c: Status 404 returned error can't find the container with id a348a602a643eaaa8acbe074517844b5a0d0e98d33be86fc4f5073fd42ed5e9c
Jul 17 13:25:19.634350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:19.634323    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c70eb6d-2caa-47c8-ac41-eea1d3e0d107" path="/var/lib/kubelet/pods/1c70eb6d-2caa-47c8-ac41-eea1d3e0d107/volumes"
Jul 17 13:25:19.913013 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:19.912964    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt" event={"ID":"f111151d-c379-4472-8c58-505e60313943","Type":"ContainerStarted","Data":"01886ef22e15d5508864b6dba26dfdf436f68d0d014c2440d799c2a33b67733a"}
Jul 17 13:25:19.913384 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:19.913060    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt" event={"ID":"f111151d-c379-4472-8c58-505e60313943","Type":"ContainerStarted","Data":"a348a602a643eaaa8acbe074517844b5a0d0e98d33be86fc4f5073fd42ed5e9c"}
Jul 17 13:25:19.929546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:19.929506    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt" podStartSLOduration=1.929493286 podStartE2EDuration="1.929493286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:25:19.928661502 +0000 UTC m=+3794.932708385" watchObservedRunningTime="2026-07-17 13:25:19.929493286 +0000 UTC m=+3794.933540224"
Jul 17 13:25:20.070010 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:20.069979    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:25:20.919133 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:20.919103    2569 generic.go:356] "Generic (PLEG): container finished" podID="f111151d-c379-4472-8c58-505e60313943" containerID="01886ef22e15d5508864b6dba26dfdf436f68d0d014c2440d799c2a33b67733a" exitCode=6
Jul 17 13:25:20.919524 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:20.919190    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt" event={"ID":"f111151d-c379-4472-8c58-505e60313943","Type":"ContainerDied","Data":"01886ef22e15d5508864b6dba26dfdf436f68d0d014c2440d799c2a33b67733a"}
Jul 17 13:25:20.919524 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:20.919318    2569 scope.go:117] "RemoveContainer" containerID="01886ef22e15d5508864b6dba26dfdf436f68d0d014c2440d799c2a33b67733a"
Jul 17 13:25:21.926249 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:21.926159    2569 generic.go:356] "Generic (PLEG): container finished" podID="f111151d-c379-4472-8c58-505e60313943" containerID="2cec48d6b243a12517e612b1923200d6ca00b38905f2afadae88492e9db2f650" exitCode=6
Jul 17 13:25:21.926249 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:21.926230    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt" event={"ID":"f111151d-c379-4472-8c58-505e60313943","Type":"ContainerDied","Data":"2cec48d6b243a12517e612b1923200d6ca00b38905f2afadae88492e9db2f650"}
Jul 17 13:25:21.926738 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:21.926294    2569 scope.go:117] "RemoveContainer" containerID="01886ef22e15d5508864b6dba26dfdf436f68d0d014c2440d799c2a33b67733a"
Jul 17 13:25:21.926738 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:21.926546    2569 scope.go:117] "RemoveContainer" containerID="2cec48d6b243a12517e612b1923200d6ca00b38905f2afadae88492e9db2f650"
Jul 17 13:25:21.926874 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:25:21.926792    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt_odh-ai-gateway-infra(f111151d-c379-4472-8c58-505e60313943)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt" podUID="f111151d-c379-4472-8c58-505e60313943"
Jul 17 13:25:22.812316 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:22.812284    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp"]
Jul 17 13:25:22.816977 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:22.816952    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp"
Jul 17 13:25:22.822949 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:22.822923    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp"]
Jul 17 13:25:22.932806 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:22.932782    2569 scope.go:117] "RemoveContainer" containerID="2cec48d6b243a12517e612b1923200d6ca00b38905f2afadae88492e9db2f650"
Jul 17 13:25:22.933118 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:25:22.932966    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt_odh-ai-gateway-infra(f111151d-c379-4472-8c58-505e60313943)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt" podUID="f111151d-c379-4472-8c58-505e60313943"
Jul 17 13:25:23.003607 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:23.003582    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac5fc92-5643-4bd7-a419-8559485945e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp\" (UID: \"6ac5fc92-5643-4bd7-a419-8559485945e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp"
Jul 17 13:25:23.105042 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:23.104977    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac5fc92-5643-4bd7-a419-8559485945e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp\" (UID: \"6ac5fc92-5643-4bd7-a419-8559485945e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp"
Jul 17 13:25:23.105610 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:23.105588    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac5fc92-5643-4bd7-a419-8559485945e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp\" (UID: \"6ac5fc92-5643-4bd7-a419-8559485945e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp"
Jul 17 13:25:23.128972 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:23.128948    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp"
Jul 17 13:25:23.260724 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:23.260697    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp"]
Jul 17 13:25:23.262878 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:25:23.262852    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac5fc92_5643_4bd7_a419_8559485945e2.slice/crio-e688ee3bbb12fa1b4d9a1e52983bd7d950d650d7b0d65e6b501ff69e45dd1d5f WatchSource:0}: Error finding container e688ee3bbb12fa1b4d9a1e52983bd7d950d650d7b0d65e6b501ff69e45dd1d5f: Status 404 returned error can't find the container with id e688ee3bbb12fa1b4d9a1e52983bd7d950d650d7b0d65e6b501ff69e45dd1d5f
Jul 17 13:25:23.938577 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:23.938550    2569 generic.go:356] "Generic (PLEG): container finished" podID="6ac5fc92-5643-4bd7-a419-8559485945e2" containerID="b05c5b602beea678b344fae1de8b3f553db16d8426ec5f30c96ecca45cb99125" exitCode=6
Jul 17 13:25:23.938924 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:23.938637    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp" event={"ID":"6ac5fc92-5643-4bd7-a419-8559485945e2","Type":"ContainerDied","Data":"b05c5b602beea678b344fae1de8b3f553db16d8426ec5f30c96ecca45cb99125"}
Jul 17 13:25:23.938924 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:23.938697    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp" event={"ID":"6ac5fc92-5643-4bd7-a419-8559485945e2","Type":"ContainerStarted","Data":"e688ee3bbb12fa1b4d9a1e52983bd7d950d650d7b0d65e6b501ff69e45dd1d5f"}
Jul 17 13:25:23.938924 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:23.938784    2569 scope.go:117] "RemoveContainer" containerID="b05c5b602beea678b344fae1de8b3f553db16d8426ec5f30c96ecca45cb99125"
Jul 17 13:25:24.946595 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:24.946520    2569 generic.go:356] "Generic (PLEG): container finished" podID="6ac5fc92-5643-4bd7-a419-8559485945e2" containerID="77c64cdd5796cd7ba0166b9cd8166e9bece995e294394b5780233a150ebd2450" exitCode=6
Jul 17 13:25:24.947003 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:24.946603    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp" event={"ID":"6ac5fc92-5643-4bd7-a419-8559485945e2","Type":"ContainerDied","Data":"77c64cdd5796cd7ba0166b9cd8166e9bece995e294394b5780233a150ebd2450"}
Jul 17 13:25:24.947003 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:24.946672    2569 scope.go:117] "RemoveContainer" containerID="b05c5b602beea678b344fae1de8b3f553db16d8426ec5f30c96ecca45cb99125"
Jul 17 13:25:24.947003 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:24.946904    2569 scope.go:117] "RemoveContainer" containerID="77c64cdd5796cd7ba0166b9cd8166e9bece995e294394b5780233a150ebd2450"
Jul 17 13:25:24.947157 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:25:24.947132    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp_odh-ai-gateway-infra(6ac5fc92-5643-4bd7-a419-8559485945e2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp" podUID="6ac5fc92-5643-4bd7-a419-8559485945e2"
Jul 17 13:25:25.952974 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:25.952938    2569 scope.go:117] "RemoveContainer" containerID="77c64cdd5796cd7ba0166b9cd8166e9bece995e294394b5780233a150ebd2450"
Jul 17 13:25:25.953392 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:25:25.953122    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp_odh-ai-gateway-infra(6ac5fc92-5643-4bd7-a419-8559485945e2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp" podUID="6ac5fc92-5643-4bd7-a419-8559485945e2"
Jul 17 13:25:31.439168 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:31.439125    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg"]
Jul 17 13:25:31.444022 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:31.443998    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg"
Jul 17 13:25:31.453125 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:31.453098    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg"]
Jul 17 13:25:31.466341 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:31.466312    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d19df20-7422-4a67-947b-c820ffe3b6d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg\" (UID: \"0d19df20-7422-4a67-947b-c820ffe3b6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg"
Jul 17 13:25:31.567440 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:31.567415    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d19df20-7422-4a67-947b-c820ffe3b6d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg\" (UID: \"0d19df20-7422-4a67-947b-c820ffe3b6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg"
Jul 17 13:25:31.568008 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:31.567988    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d19df20-7422-4a67-947b-c820ffe3b6d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg\" (UID: \"0d19df20-7422-4a67-947b-c820ffe3b6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg"
Jul 17 13:25:31.757488 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:31.757463    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg"
Jul 17 13:25:31.888627 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:31.888481    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg"]
Jul 17 13:25:31.890868 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:25:31.890840    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d19df20_7422_4a67_947b_c820ffe3b6d7.slice/crio-a7bb6d73c1d98575ce0ffa6da8cc0323050aa911b484d69fe148d6783243b89b WatchSource:0}: Error finding container a7bb6d73c1d98575ce0ffa6da8cc0323050aa911b484d69fe148d6783243b89b: Status 404 returned error can't find the container with id a7bb6d73c1d98575ce0ffa6da8cc0323050aa911b484d69fe148d6783243b89b
Jul 17 13:25:31.980333 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:31.980306    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg" event={"ID":"0d19df20-7422-4a67-947b-c820ffe3b6d7","Type":"ContainerStarted","Data":"a7bb6d73c1d98575ce0ffa6da8cc0323050aa911b484d69fe148d6783243b89b"}
Jul 17 13:25:32.986881 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:32.986849    2569 generic.go:356] "Generic (PLEG): container finished" podID="0d19df20-7422-4a67-947b-c820ffe3b6d7" containerID="22b05e02504d0ab59ce574bf4a2757599597297d4cf03e487103694c14029b3d" exitCode=6
Jul 17 13:25:32.987257 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:32.986930    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg" event={"ID":"0d19df20-7422-4a67-947b-c820ffe3b6d7","Type":"ContainerDied","Data":"22b05e02504d0ab59ce574bf4a2757599597297d4cf03e487103694c14029b3d"}
Jul 17 13:25:32.987257 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:32.987106    2569 scope.go:117] "RemoveContainer" containerID="22b05e02504d0ab59ce574bf4a2757599597297d4cf03e487103694c14029b3d"
Jul 17 13:25:33.628613 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:33.628585    2569 scope.go:117] "RemoveContainer" containerID="2cec48d6b243a12517e612b1923200d6ca00b38905f2afadae88492e9db2f650"
Jul 17 13:25:33.993555 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:33.993523    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt" event={"ID":"f111151d-c379-4472-8c58-505e60313943","Type":"ContainerStarted","Data":"d123cf6dda8a97f18debfdd580e6b3381e09d42917e630f0d17d26326bd9e488"}
Jul 17 13:25:33.995377 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:33.995352    2569 generic.go:356] "Generic (PLEG): container finished" podID="0d19df20-7422-4a67-947b-c820ffe3b6d7" containerID="9c69d90022b8f74e6dc8529013c777f2393433efe085229b3124b79a5de7967e" exitCode=6
Jul 17 13:25:33.995488 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:33.995438    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg" event={"ID":"0d19df20-7422-4a67-947b-c820ffe3b6d7","Type":"ContainerDied","Data":"9c69d90022b8f74e6dc8529013c777f2393433efe085229b3124b79a5de7967e"}
Jul 17 13:25:33.995537 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:33.995502    2569 scope.go:117] "RemoveContainer" containerID="22b05e02504d0ab59ce574bf4a2757599597297d4cf03e487103694c14029b3d"
Jul 17 13:25:33.995610 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:33.995595    2569 scope.go:117] "RemoveContainer" containerID="9c69d90022b8f74e6dc8529013c777f2393433efe085229b3124b79a5de7967e"
Jul 17 13:25:33.995800 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:25:33.995782    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg_odh-ai-gateway-infra(0d19df20-7422-4a67-947b-c820ffe3b6d7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg" podUID="0d19df20-7422-4a67-947b-c820ffe3b6d7"
Jul 17 13:25:35.002599 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:35.002569    2569 generic.go:356] "Generic (PLEG): container finished" podID="f111151d-c379-4472-8c58-505e60313943" containerID="d123cf6dda8a97f18debfdd580e6b3381e09d42917e630f0d17d26326bd9e488" exitCode=6
Jul 17 13:25:35.003040 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:35.002630    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt" event={"ID":"f111151d-c379-4472-8c58-505e60313943","Type":"ContainerDied","Data":"d123cf6dda8a97f18debfdd580e6b3381e09d42917e630f0d17d26326bd9e488"}
Jul 17 13:25:35.003040 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:35.002693    2569 scope.go:117] "RemoveContainer" containerID="2cec48d6b243a12517e612b1923200d6ca00b38905f2afadae88492e9db2f650"
Jul 17 13:25:35.003040 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:35.003017    2569 scope.go:117] "RemoveContainer" containerID="d123cf6dda8a97f18debfdd580e6b3381e09d42917e630f0d17d26326bd9e488"
Jul 17 13:25:35.003272 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:25:35.003238    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt_odh-ai-gateway-infra(f111151d-c379-4472-8c58-505e60313943)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt" podUID="f111151d-c379-4472-8c58-505e60313943"
Jul 17 13:25:35.004846 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:35.004829    2569 scope.go:117] "RemoveContainer" containerID="9c69d90022b8f74e6dc8529013c777f2393433efe085229b3124b79a5de7967e"
Jul 17 13:25:35.004984 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:25:35.004967    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg_odh-ai-gateway-infra(0d19df20-7422-4a67-947b-c820ffe3b6d7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg" podUID="0d19df20-7422-4a67-947b-c820ffe3b6d7"
Jul 17 13:25:35.029006 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:35.028982    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt"]
Jul 17 13:25:36.145883 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:36.145853    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt"
Jul 17 13:25:36.202905 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:36.202880    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f111151d-c379-4472-8c58-505e60313943-maas-api-service-ca\") pod \"f111151d-c379-4472-8c58-505e60313943\" (UID: \"f111151d-c379-4472-8c58-505e60313943\") "
Jul 17 13:25:36.203215 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:36.203196    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f111151d-c379-4472-8c58-505e60313943-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f111151d-c379-4472-8c58-505e60313943" (UID: "f111151d-c379-4472-8c58-505e60313943"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:25:36.303565 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:36.303505    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f111151d-c379-4472-8c58-505e60313943-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:25:37.016366 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.016337    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt"
Jul 17 13:25:37.016541 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.016362    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt" event={"ID":"f111151d-c379-4472-8c58-505e60313943","Type":"ContainerDied","Data":"a348a602a643eaaa8acbe074517844b5a0d0e98d33be86fc4f5073fd42ed5e9c"}
Jul 17 13:25:37.016541 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.016423    2569 scope.go:117] "RemoveContainer" containerID="d123cf6dda8a97f18debfdd580e6b3381e09d42917e630f0d17d26326bd9e488"
Jul 17 13:25:37.037594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.037552    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt"]
Jul 17 13:25:37.043722 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.043697    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-k95bt"]
Jul 17 13:25:37.277502 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.272780    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"]
Jul 17 13:25:37.277502 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.273493    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f111151d-c379-4472-8c58-505e60313943" containerName="revoke-keys"
Jul 17 13:25:37.277502 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.273512    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f111151d-c379-4472-8c58-505e60313943" containerName="revoke-keys"
Jul 17 13:25:37.277502 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.273543    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f111151d-c379-4472-8c58-505e60313943" containerName="revoke-keys"
Jul 17 13:25:37.277502 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.273551    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f111151d-c379-4472-8c58-505e60313943" containerName="revoke-keys"
Jul 17 13:25:37.277502 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.273690    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f111151d-c379-4472-8c58-505e60313943" containerName="revoke-keys"
Jul 17 13:25:37.277502 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.273703    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f111151d-c379-4472-8c58-505e60313943" containerName="revoke-keys"
Jul 17 13:25:37.279818 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.279796    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"
Jul 17 13:25:37.282547 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.282520    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lwvzj\""
Jul 17 13:25:37.282662 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.282649    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-771703-serving-cert\""
Jul 17 13:25:37.283166 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.283080    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 13:25:37.285926 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.285906    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"]
Jul 17 13:25:37.311831 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.311805    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-maas-api-tls\") pod \"maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd\" (UID: \"b1b016ab-a1bb-4135-9d3e-afebdcb4d690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"
Jul 17 13:25:37.311936 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.311835    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-openshift-service-ca\") pod \"maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd\" (UID: \"b1b016ab-a1bb-4135-9d3e-afebdcb4d690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"
Jul 17 13:25:37.311936 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.311866    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pd92\" (UniqueName: \"kubernetes.io/projected/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-kube-api-access-9pd92\") pod \"maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd\" (UID: \"b1b016ab-a1bb-4135-9d3e-afebdcb4d690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"
Jul 17 13:25:37.413261 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.413236    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pd92\" (UniqueName: \"kubernetes.io/projected/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-kube-api-access-9pd92\") pod \"maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd\" (UID: \"b1b016ab-a1bb-4135-9d3e-afebdcb4d690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"
Jul 17 13:25:37.413361 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.413337    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-maas-api-tls\") pod \"maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd\" (UID: \"b1b016ab-a1bb-4135-9d3e-afebdcb4d690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"
Jul 17 13:25:37.413411 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.413365    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-openshift-service-ca\") pod \"maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd\" (UID: \"b1b016ab-a1bb-4135-9d3e-afebdcb4d690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"
Jul 17 13:25:37.413487 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:25:37.413471    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-771703-serving-cert: secret "maas-api-e2e-api-a-771703-serving-cert" not found
Jul 17 13:25:37.413573 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:25:37.413562    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-maas-api-tls podName:b1b016ab-a1bb-4135-9d3e-afebdcb4d690 nodeName:}" failed. No retries permitted until 2026-07-17 13:25:37.913542822 +0000 UTC m=+3812.917589685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-maas-api-tls") pod "maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd" (UID: "b1b016ab-a1bb-4135-9d3e-afebdcb4d690") : secret "maas-api-e2e-api-a-771703-serving-cert" not found
Jul 17 13:25:37.414084 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.414066    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-openshift-service-ca\") pod \"maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd\" (UID: \"b1b016ab-a1bb-4135-9d3e-afebdcb4d690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"
Jul 17 13:25:37.425425 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.425406    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pd92\" (UniqueName: \"kubernetes.io/projected/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-kube-api-access-9pd92\") pod \"maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd\" (UID: \"b1b016ab-a1bb-4135-9d3e-afebdcb4d690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"
Jul 17 13:25:37.628459 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.628403    2569 scope.go:117] "RemoveContainer" containerID="77c64cdd5796cd7ba0166b9cd8166e9bece995e294394b5780233a150ebd2450"
Jul 17 13:25:37.634950 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.634932    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f111151d-c379-4472-8c58-505e60313943" path="/var/lib/kubelet/pods/f111151d-c379-4472-8c58-505e60313943/volumes"
Jul 17 13:25:37.917523 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.917467    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-maas-api-tls\") pod \"maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd\" (UID: \"b1b016ab-a1bb-4135-9d3e-afebdcb4d690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"
Jul 17 13:25:37.919661 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:37.919634    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-maas-api-tls\") pod \"maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd\" (UID: \"b1b016ab-a1bb-4135-9d3e-afebdcb4d690\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"
Jul 17 13:25:38.023701 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:38.023666    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp" event={"ID":"6ac5fc92-5643-4bd7-a419-8559485945e2","Type":"ContainerStarted","Data":"05ce1545315e9d7d8b3fc377d2443524cc00011ec6de2ec1c1e77195cb7cf263"}
Jul 17 13:25:38.192250 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:38.192137    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"
Jul 17 13:25:38.325253 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:38.325207    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp" podStartSLOduration=16.325190141 podStartE2EDuration="16.325190141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:25:38.057094213 +0000 UTC m=+3813.061141095" watchObservedRunningTime="2026-07-17 13:25:38.325190141 +0000 UTC m=+3813.329237023"
Jul 17 13:25:38.326255 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:38.326233    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"]
Jul 17 13:25:38.327907 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:25:38.327882    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b016ab_a1bb_4135_9d3e_afebdcb4d690.slice/crio-1e160400b3c3c5082d4c7c77d27d281a0676779767f73232b8eeef4bea065fb7 WatchSource:0}: Error finding container 1e160400b3c3c5082d4c7c77d27d281a0676779767f73232b8eeef4bea065fb7: Status 404 returned error can't find the container with id 1e160400b3c3c5082d4c7c77d27d281a0676779767f73232b8eeef4bea065fb7
Jul 17 13:25:38.329289 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:38.329272    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 13:25:39.031521 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:39.031330    2569 generic.go:356] "Generic (PLEG): container finished" podID="6ac5fc92-5643-4bd7-a419-8559485945e2" containerID="05ce1545315e9d7d8b3fc377d2443524cc00011ec6de2ec1c1e77195cb7cf263" exitCode=6
Jul 17 13:25:39.031521 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:39.031491    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp" event={"ID":"6ac5fc92-5643-4bd7-a419-8559485945e2","Type":"ContainerDied","Data":"05ce1545315e9d7d8b3fc377d2443524cc00011ec6de2ec1c1e77195cb7cf263"}
Jul 17 13:25:39.031776 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:39.031551    2569 scope.go:117] "RemoveContainer" containerID="77c64cdd5796cd7ba0166b9cd8166e9bece995e294394b5780233a150ebd2450"
Jul 17 13:25:39.031931 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:39.031911    2569 scope.go:117] "RemoveContainer" containerID="05ce1545315e9d7d8b3fc377d2443524cc00011ec6de2ec1c1e77195cb7cf263"
Jul 17 13:25:39.032100 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:25:39.032082    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp_odh-ai-gateway-infra(6ac5fc92-5643-4bd7-a419-8559485945e2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp" podUID="6ac5fc92-5643-4bd7-a419-8559485945e2"
Jul 17 13:25:39.033360 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:39.033332    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd" event={"ID":"b1b016ab-a1bb-4135-9d3e-afebdcb4d690","Type":"ContainerStarted","Data":"1e160400b3c3c5082d4c7c77d27d281a0676779767f73232b8eeef4bea065fb7"}
Jul 17 13:25:39.070086 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:39.070060    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp"]
Jul 17 13:25:40.039392 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:40.039357    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd" event={"ID":"b1b016ab-a1bb-4135-9d3e-afebdcb4d690","Type":"ContainerStarted","Data":"ef1734b48e9a4d020cd231c76b26da6b98392c5b3709c1793e842e7f8fd76dac"}
Jul 17 13:25:40.039870 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:40.039422    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"
Jul 17 13:25:40.057607 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:40.057534    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd" podStartSLOduration=2.350714158 podStartE2EDuration="3.057518901s" totalImagesPullingTime="706.804743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:25:40.056360181 +0000 UTC m=+3815.060407066" watchObservedRunningTime="2026-07-17 13:25:40.057518901 +0000 UTC m=+3815.061565805"
Jul 17 13:25:40.177159 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:40.177137    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp"
Jul 17 13:25:40.236268 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:40.236240    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac5fc92-5643-4bd7-a419-8559485945e2-maas-api-service-ca\") pod \"6ac5fc92-5643-4bd7-a419-8559485945e2\" (UID: \"6ac5fc92-5643-4bd7-a419-8559485945e2\") "
Jul 17 13:25:40.236553 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:40.236530    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac5fc92-5643-4bd7-a419-8559485945e2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ac5fc92-5643-4bd7-a419-8559485945e2" (UID: "6ac5fc92-5643-4bd7-a419-8559485945e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:25:40.337554 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:40.337503    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac5fc92-5643-4bd7-a419-8559485945e2-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:25:41.048336 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:41.048307    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp"
Jul 17 13:25:41.048788 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:41.048351    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp" event={"ID":"6ac5fc92-5643-4bd7-a419-8559485945e2","Type":"ContainerDied","Data":"e688ee3bbb12fa1b4d9a1e52983bd7d950d650d7b0d65e6b501ff69e45dd1d5f"}
Jul 17 13:25:41.048788 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:41.048422    2569 scope.go:117] "RemoveContainer" containerID="05ce1545315e9d7d8b3fc377d2443524cc00011ec6de2ec1c1e77195cb7cf263"
Jul 17 13:25:41.069948 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:41.069925    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp"]
Jul 17 13:25:41.075587 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:41.075565    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlhxp"]
Jul 17 13:25:41.633820 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:41.633797    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ac5fc92-5643-4bd7-a419-8559485945e2" path="/var/lib/kubelet/pods/6ac5fc92-5643-4bd7-a419-8559485945e2/volumes"
Jul 17 13:25:42.068467 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:42.068438    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:25:45.155091 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.155053    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"]
Jul 17 13:25:45.155507 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.155482    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ac5fc92-5643-4bd7-a419-8559485945e2" containerName="revoke-keys"
Jul 17 13:25:45.155507 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.155497    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ac5fc92-5643-4bd7-a419-8559485945e2" containerName="revoke-keys"
Jul 17 13:25:45.155507 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.155508    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ac5fc92-5643-4bd7-a419-8559485945e2" containerName="revoke-keys"
Jul 17 13:25:45.155680 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.155517    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ac5fc92-5643-4bd7-a419-8559485945e2" containerName="revoke-keys"
Jul 17 13:25:45.155680 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.155534    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f111151d-c379-4472-8c58-505e60313943" containerName="revoke-keys"
Jul 17 13:25:45.155680 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.155540    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f111151d-c379-4472-8c58-505e60313943" containerName="revoke-keys"
Jul 17 13:25:45.155680 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.155620    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ac5fc92-5643-4bd7-a419-8559485945e2" containerName="revoke-keys"
Jul 17 13:25:45.155680 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.155629    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ac5fc92-5643-4bd7-a419-8559485945e2" containerName="revoke-keys"
Jul 17 13:25:45.155680 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.155636    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f111151d-c379-4472-8c58-505e60313943" containerName="revoke-keys"
Jul 17 13:25:45.159743 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.159725    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"
Jul 17 13:25:45.162931 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.162912    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6fa40d-serving-cert\""
Jul 17 13:25:45.168915 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.168897    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"]
Jul 17 13:25:45.278360 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.274243    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd91ba59-2a5a-48c0-a791-1def0a6762b8-openshift-service-ca\") pod \"maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59\" (UID: \"fd91ba59-2a5a-48c0-a791-1def0a6762b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"
Jul 17 13:25:45.278360 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.274349    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rsr8\" (UniqueName: \"kubernetes.io/projected/fd91ba59-2a5a-48c0-a791-1def0a6762b8-kube-api-access-8rsr8\") pod \"maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59\" (UID: \"fd91ba59-2a5a-48c0-a791-1def0a6762b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"
Jul 17 13:25:45.278360 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.274389    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd91ba59-2a5a-48c0-a791-1def0a6762b8-maas-api-tls\") pod \"maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59\" (UID: \"fd91ba59-2a5a-48c0-a791-1def0a6762b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"
Jul 17 13:25:45.375029 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.375001    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rsr8\" (UniqueName: \"kubernetes.io/projected/fd91ba59-2a5a-48c0-a791-1def0a6762b8-kube-api-access-8rsr8\") pod \"maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59\" (UID: \"fd91ba59-2a5a-48c0-a791-1def0a6762b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"
Jul 17 13:25:45.375137 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.375036    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd91ba59-2a5a-48c0-a791-1def0a6762b8-maas-api-tls\") pod \"maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59\" (UID: \"fd91ba59-2a5a-48c0-a791-1def0a6762b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"
Jul 17 13:25:45.375189 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:25:45.375138    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-serving-cert: secret "maas-api-e2e-api-b-6fa40d-serving-cert" not found
Jul 17 13:25:45.375189 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.375170    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd91ba59-2a5a-48c0-a791-1def0a6762b8-openshift-service-ca\") pod \"maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59\" (UID: \"fd91ba59-2a5a-48c0-a791-1def0a6762b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"
Jul 17 13:25:45.375189 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:25:45.375184    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd91ba59-2a5a-48c0-a791-1def0a6762b8-maas-api-tls podName:fd91ba59-2a5a-48c0-a791-1def0a6762b8 nodeName:}" failed. No retries permitted until 2026-07-17 13:25:45.875169778 +0000 UTC m=+3820.879216638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd91ba59-2a5a-48c0-a791-1def0a6762b8-maas-api-tls") pod "maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59" (UID: "fd91ba59-2a5a-48c0-a791-1def0a6762b8") : secret "maas-api-e2e-api-b-6fa40d-serving-cert" not found
Jul 17 13:25:45.375679 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.375659    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd91ba59-2a5a-48c0-a791-1def0a6762b8-openshift-service-ca\") pod \"maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59\" (UID: \"fd91ba59-2a5a-48c0-a791-1def0a6762b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"
Jul 17 13:25:45.383887 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.383868    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rsr8\" (UniqueName: \"kubernetes.io/projected/fd91ba59-2a5a-48c0-a791-1def0a6762b8-kube-api-access-8rsr8\") pod \"maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59\" (UID: \"fd91ba59-2a5a-48c0-a791-1def0a6762b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"
Jul 17 13:25:45.632276 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.632221    2569 scope.go:117] "RemoveContainer" containerID="9c69d90022b8f74e6dc8529013c777f2393433efe085229b3124b79a5de7967e"
Jul 17 13:25:45.879613 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.879573    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd91ba59-2a5a-48c0-a791-1def0a6762b8-maas-api-tls\") pod \"maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59\" (UID: \"fd91ba59-2a5a-48c0-a791-1def0a6762b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"
Jul 17 13:25:45.881694 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:45.881675    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd91ba59-2a5a-48c0-a791-1def0a6762b8-maas-api-tls\") pod \"maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59\" (UID: \"fd91ba59-2a5a-48c0-a791-1def0a6762b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"
Jul 17 13:25:46.054813 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:46.054793    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"
Jul 17 13:25:46.071139 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:46.071114    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"
Jul 17 13:25:46.074291 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:46.074266    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg" event={"ID":"0d19df20-7422-4a67-947b-c820ffe3b6d7","Type":"ContainerStarted","Data":"8d59e5597a6866aab7aa496a283a6b29ea120e8882be97a254f011e6eb28cbcd"}
Jul 17 13:25:46.093782 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:46.093725    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg" podStartSLOduration=15.093710245 podStartE2EDuration="15.093710245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:25:46.091201459 +0000 UTC m=+3821.095248342" watchObservedRunningTime="2026-07-17 13:25:46.093710245 +0000 UTC m=+3821.097757126"
Jul 17 13:25:46.207486 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:46.207452    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"]
Jul 17 13:25:46.209683 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:25:46.209647    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd91ba59_2a5a_48c0_a791_1def0a6762b8.slice/crio-8d2536f7d7ed2f136a15c91c46c03a579c34264c4ac02ea9e904e35b47c5a8a8 WatchSource:0}: Error finding container 8d2536f7d7ed2f136a15c91c46c03a579c34264c4ac02ea9e904e35b47c5a8a8: Status 404 returned error can't find the container with id 8d2536f7d7ed2f136a15c91c46c03a579c34264c4ac02ea9e904e35b47c5a8a8
Jul 17 13:25:47.080047 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:47.080016    2569 generic.go:356] "Generic (PLEG): container finished" podID="0d19df20-7422-4a67-947b-c820ffe3b6d7" containerID="8d59e5597a6866aab7aa496a283a6b29ea120e8882be97a254f011e6eb28cbcd" exitCode=6
Jul 17 13:25:47.080212 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:47.080085    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg" event={"ID":"0d19df20-7422-4a67-947b-c820ffe3b6d7","Type":"ContainerDied","Data":"8d59e5597a6866aab7aa496a283a6b29ea120e8882be97a254f011e6eb28cbcd"}
Jul 17 13:25:47.080212 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:47.080140    2569 scope.go:117] "RemoveContainer" containerID="9c69d90022b8f74e6dc8529013c777f2393433efe085229b3124b79a5de7967e"
Jul 17 13:25:47.080439 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:47.080419    2569 scope.go:117] "RemoveContainer" containerID="8d59e5597a6866aab7aa496a283a6b29ea120e8882be97a254f011e6eb28cbcd"
Jul 17 13:25:47.080648 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:25:47.080628    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg_odh-ai-gateway-infra(0d19df20-7422-4a67-947b-c820ffe3b6d7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg" podUID="0d19df20-7422-4a67-947b-c820ffe3b6d7"
Jul 17 13:25:47.081683 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:47.081661    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59" event={"ID":"fd91ba59-2a5a-48c0-a791-1def0a6762b8","Type":"ContainerStarted","Data":"310cf33dec08593c81cd886a4c996a7770490e35fbb4c5ecc2a8d926d43c9297"}
Jul 17 13:25:47.081801 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:47.081712    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"
Jul 17 13:25:47.081801 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:47.081737    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59" event={"ID":"fd91ba59-2a5a-48c0-a791-1def0a6762b8","Type":"ContainerStarted","Data":"8d2536f7d7ed2f136a15c91c46c03a579c34264c4ac02ea9e904e35b47c5a8a8"}
Jul 17 13:25:47.107284 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:47.107261    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg"]
Jul 17 13:25:47.166231 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:47.166190    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59" podStartSLOduration=1.6576759650000001 podStartE2EDuration="2.166178704s" totalImagesPullingTime="508.502739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:25:47.163462957 +0000 UTC m=+3822.167509838" watchObservedRunningTime="2026-07-17 13:25:47.166178704 +0000 UTC m=+3822.170225587"
Jul 17 13:25:48.216167 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:48.216140    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg"
Jul 17 13:25:48.302281 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:48.302255    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d19df20-7422-4a67-947b-c820ffe3b6d7-maas-api-service-ca\") pod \"0d19df20-7422-4a67-947b-c820ffe3b6d7\" (UID: \"0d19df20-7422-4a67-947b-c820ffe3b6d7\") "
Jul 17 13:25:48.302592 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:48.302570    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d19df20-7422-4a67-947b-c820ffe3b6d7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0d19df20-7422-4a67-947b-c820ffe3b6d7" (UID: "0d19df20-7422-4a67-947b-c820ffe3b6d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:25:48.403092 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:48.403031    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d19df20-7422-4a67-947b-c820ffe3b6d7-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:25:49.096790 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:49.096735    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg"
Jul 17 13:25:49.096790 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:49.096768    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg" event={"ID":"0d19df20-7422-4a67-947b-c820ffe3b6d7","Type":"ContainerDied","Data":"a7bb6d73c1d98575ce0ffa6da8cc0323050aa911b484d69fe148d6783243b89b"}
Jul 17 13:25:49.096982 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:49.096850    2569 scope.go:117] "RemoveContainer" containerID="8d59e5597a6866aab7aa496a283a6b29ea120e8882be97a254f011e6eb28cbcd"
Jul 17 13:25:49.119295 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:49.119268    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg"]
Jul 17 13:25:49.122685 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:49.122660    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-q6vrg"]
Jul 17 13:25:49.635051 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:49.635020    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d19df20-7422-4a67-947b-c820ffe3b6d7" path="/var/lib/kubelet/pods/0d19df20-7422-4a67-947b-c820ffe3b6d7/volumes"
Jul 17 13:25:52.969837 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:52.969795    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:25:53.094655 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:25:53.094627    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"
Jul 17 13:26:08.415098 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.415064    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"]
Jul 17 13:26:08.415625 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.415311    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd" podUID="b1b016ab-a1bb-4135-9d3e-afebdcb4d690" containerName="maas-api" containerID="cri-o://ef1734b48e9a4d020cd231c76b26da6b98392c5b3709c1793e842e7f8fd76dac" gracePeriod=30
Jul 17 13:26:08.661673 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.661647    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"
Jul 17 13:26:08.745666 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.745642    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-openshift-service-ca\") pod \"b1b016ab-a1bb-4135-9d3e-afebdcb4d690\" (UID: \"b1b016ab-a1bb-4135-9d3e-afebdcb4d690\") "
Jul 17 13:26:08.745813 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.745689    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pd92\" (UniqueName: \"kubernetes.io/projected/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-kube-api-access-9pd92\") pod \"b1b016ab-a1bb-4135-9d3e-afebdcb4d690\" (UID: \"b1b016ab-a1bb-4135-9d3e-afebdcb4d690\") "
Jul 17 13:26:08.745888 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.745824    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-maas-api-tls\") pod \"b1b016ab-a1bb-4135-9d3e-afebdcb4d690\" (UID: \"b1b016ab-a1bb-4135-9d3e-afebdcb4d690\") "
Jul 17 13:26:08.746010 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.745988    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b1b016ab-a1bb-4135-9d3e-afebdcb4d690" (UID: "b1b016ab-a1bb-4135-9d3e-afebdcb4d690"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:26:08.746136 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.746119    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-openshift-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:26:08.747614 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.747593    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-kube-api-access-9pd92" (OuterVolumeSpecName: "kube-api-access-9pd92") pod "b1b016ab-a1bb-4135-9d3e-afebdcb4d690" (UID: "b1b016ab-a1bb-4135-9d3e-afebdcb4d690"). InnerVolumeSpecName "kube-api-access-9pd92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:26:08.747741 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.747726    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b1b016ab-a1bb-4135-9d3e-afebdcb4d690" (UID: "b1b016ab-a1bb-4135-9d3e-afebdcb4d690"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:26:08.846917 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.846893    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-maas-api-tls\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:26:08.846917 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.846912    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pd92\" (UniqueName: \"kubernetes.io/projected/b1b016ab-a1bb-4135-9d3e-afebdcb4d690-kube-api-access-9pd92\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:26:08.959044 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.959013    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr"]
Jul 17 13:26:08.959375 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.959358    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ac5fc92-5643-4bd7-a419-8559485945e2" containerName="revoke-keys"
Jul 17 13:26:08.959456 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.959377    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ac5fc92-5643-4bd7-a419-8559485945e2" containerName="revoke-keys"
Jul 17 13:26:08.959456 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.959395    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d19df20-7422-4a67-947b-c820ffe3b6d7" containerName="revoke-keys"
Jul 17 13:26:08.959456 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.959404    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d19df20-7422-4a67-947b-c820ffe3b6d7" containerName="revoke-keys"
Jul 17 13:26:08.959456 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.959422    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1b016ab-a1bb-4135-9d3e-afebdcb4d690" containerName="maas-api"
Jul 17 13:26:08.959456 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.959432    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1b016ab-a1bb-4135-9d3e-afebdcb4d690" containerName="maas-api"
Jul 17 13:26:08.959456 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.959442    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d19df20-7422-4a67-947b-c820ffe3b6d7" containerName="revoke-keys"
Jul 17 13:26:08.959456 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.959450    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d19df20-7422-4a67-947b-c820ffe3b6d7" containerName="revoke-keys"
Jul 17 13:26:08.959835 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.959546    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d19df20-7422-4a67-947b-c820ffe3b6d7" containerName="revoke-keys"
Jul 17 13:26:08.959835 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.959558    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ac5fc92-5643-4bd7-a419-8559485945e2" containerName="revoke-keys"
Jul 17 13:26:08.959835 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.959569    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d19df20-7422-4a67-947b-c820ffe3b6d7" containerName="revoke-keys"
Jul 17 13:26:08.959835 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.959578    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1b016ab-a1bb-4135-9d3e-afebdcb4d690" containerName="maas-api"
Jul 17 13:26:08.962800 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.962782    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr"
Jul 17 13:26:08.965316 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.965298    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:26:08.969532 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:08.969510    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr"]
Jul 17 13:26:09.048418 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:09.048358    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86e3408e-a2d2-482c-837e-5dc77aa3c6b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr\" (UID: \"86e3408e-a2d2-482c-837e-5dc77aa3c6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr"
Jul 17 13:26:09.149459 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:09.149432    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86e3408e-a2d2-482c-837e-5dc77aa3c6b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr\" (UID: \"86e3408e-a2d2-482c-837e-5dc77aa3c6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr"
Jul 17 13:26:09.150020 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:09.150002    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86e3408e-a2d2-482c-837e-5dc77aa3c6b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr\" (UID: \"86e3408e-a2d2-482c-837e-5dc77aa3c6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr"
Jul 17 13:26:09.197671 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:09.197648    2569 generic.go:356] "Generic (PLEG): container finished" podID="b1b016ab-a1bb-4135-9d3e-afebdcb4d690" containerID="ef1734b48e9a4d020cd231c76b26da6b98392c5b3709c1793e842e7f8fd76dac" exitCode=0
Jul 17 13:26:09.197778 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:09.197677    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd" event={"ID":"b1b016ab-a1bb-4135-9d3e-afebdcb4d690","Type":"ContainerDied","Data":"ef1734b48e9a4d020cd231c76b26da6b98392c5b3709c1793e842e7f8fd76dac"}
Jul 17 13:26:09.197778 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:09.197726    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd" event={"ID":"b1b016ab-a1bb-4135-9d3e-afebdcb4d690","Type":"ContainerDied","Data":"1e160400b3c3c5082d4c7c77d27d281a0676779767f73232b8eeef4bea065fb7"}
Jul 17 13:26:09.197778 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:09.197688    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"
Jul 17 13:26:09.197901 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:09.197763    2569 scope.go:117] "RemoveContainer" containerID="ef1734b48e9a4d020cd231c76b26da6b98392c5b3709c1793e842e7f8fd76dac"
Jul 17 13:26:09.206406 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:09.206389    2569 scope.go:117] "RemoveContainer" containerID="ef1734b48e9a4d020cd231c76b26da6b98392c5b3709c1793e842e7f8fd76dac"
Jul 17 13:26:09.206638 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:26:09.206621    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef1734b48e9a4d020cd231c76b26da6b98392c5b3709c1793e842e7f8fd76dac\": container with ID starting with ef1734b48e9a4d020cd231c76b26da6b98392c5b3709c1793e842e7f8fd76dac not found: ID does not exist" containerID="ef1734b48e9a4d020cd231c76b26da6b98392c5b3709c1793e842e7f8fd76dac"
Jul 17 13:26:09.206682 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:09.206645    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef1734b48e9a4d020cd231c76b26da6b98392c5b3709c1793e842e7f8fd76dac"} err="failed to get container status \"ef1734b48e9a4d020cd231c76b26da6b98392c5b3709c1793e842e7f8fd76dac\": rpc error: code = NotFound desc = could not find container \"ef1734b48e9a4d020cd231c76b26da6b98392c5b3709c1793e842e7f8fd76dac\": container with ID starting with ef1734b48e9a4d020cd231c76b26da6b98392c5b3709c1793e842e7f8fd76dac not found: ID does not exist"
Jul 17 13:26:09.220307 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:09.220263    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"]
Jul 17 13:26:09.223120 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:09.223099    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-771703-6fd96f4c5-ppjfd"]
Jul 17 13:26:09.274991 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:09.274969    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr"
Jul 17 13:26:09.396980 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:09.396959    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr"]
Jul 17 13:26:09.399495 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:26:09.399468    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e3408e_a2d2_482c_837e_5dc77aa3c6b4.slice/crio-65b7a383c11cf3e86a1925f71006f989fae09663708de351883d11434bb11f85 WatchSource:0}: Error finding container 65b7a383c11cf3e86a1925f71006f989fae09663708de351883d11434bb11f85: Status 404 returned error can't find the container with id 65b7a383c11cf3e86a1925f71006f989fae09663708de351883d11434bb11f85
Jul 17 13:26:09.633237 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:09.633181    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1b016ab-a1bb-4135-9d3e-afebdcb4d690" path="/var/lib/kubelet/pods/b1b016ab-a1bb-4135-9d3e-afebdcb4d690/volumes"
Jul 17 13:26:10.205571 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:10.205543    2569 generic.go:356] "Generic (PLEG): container finished" podID="86e3408e-a2d2-482c-837e-5dc77aa3c6b4" containerID="a6f03fbef147f12a3fa5396543da01b640128a2449c956c75a10c403b1f2eab3" exitCode=6
Jul 17 13:26:10.205682 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:10.205594    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr" event={"ID":"86e3408e-a2d2-482c-837e-5dc77aa3c6b4","Type":"ContainerDied","Data":"a6f03fbef147f12a3fa5396543da01b640128a2449c956c75a10c403b1f2eab3"}
Jul 17 13:26:10.205682 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:10.205629    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr" event={"ID":"86e3408e-a2d2-482c-837e-5dc77aa3c6b4","Type":"ContainerStarted","Data":"65b7a383c11cf3e86a1925f71006f989fae09663708de351883d11434bb11f85"}
Jul 17 13:26:10.205816 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:10.205803    2569 scope.go:117] "RemoveContainer" containerID="a6f03fbef147f12a3fa5396543da01b640128a2449c956c75a10c403b1f2eab3"
Jul 17 13:26:11.211941 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:11.211873    2569 generic.go:356] "Generic (PLEG): container finished" podID="86e3408e-a2d2-482c-837e-5dc77aa3c6b4" containerID="3408c25eaceae7a34a7c98f0e522ac0abf9592d192ec8c8afa6ef20124ecca73" exitCode=6
Jul 17 13:26:11.212333 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:11.211969    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr" event={"ID":"86e3408e-a2d2-482c-837e-5dc77aa3c6b4","Type":"ContainerDied","Data":"3408c25eaceae7a34a7c98f0e522ac0abf9592d192ec8c8afa6ef20124ecca73"}
Jul 17 13:26:11.212333 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:11.212040    2569 scope.go:117] "RemoveContainer" containerID="a6f03fbef147f12a3fa5396543da01b640128a2449c956c75a10c403b1f2eab3"
Jul 17 13:26:11.212333 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:11.212131    2569 scope.go:117] "RemoveContainer" containerID="3408c25eaceae7a34a7c98f0e522ac0abf9592d192ec8c8afa6ef20124ecca73"
Jul 17 13:26:11.212333 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:26:11.212304    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr_odh-ai-gateway-infra(86e3408e-a2d2-482c-837e-5dc77aa3c6b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr" podUID="86e3408e-a2d2-482c-837e-5dc77aa3c6b4"
Jul 17 13:26:11.970142 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:11.970107    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:26:12.219917 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:12.219889    2569 scope.go:117] "RemoveContainer" containerID="3408c25eaceae7a34a7c98f0e522ac0abf9592d192ec8c8afa6ef20124ecca73"
Jul 17 13:26:12.220313 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:26:12.220086    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr_odh-ai-gateway-infra(86e3408e-a2d2-482c-837e-5dc77aa3c6b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr" podUID="86e3408e-a2d2-482c-837e-5dc77aa3c6b4"
Jul 17 13:26:20.247470 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:20.247423    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n"]
Jul 17 13:26:20.248002 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:20.247966    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d19df20-7422-4a67-947b-c820ffe3b6d7" containerName="revoke-keys"
Jul 17 13:26:20.248002 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:20.247988    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d19df20-7422-4a67-947b-c820ffe3b6d7" containerName="revoke-keys"
Jul 17 13:26:20.248203 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:20.248127    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d19df20-7422-4a67-947b-c820ffe3b6d7" containerName="revoke-keys"
Jul 17 13:26:20.252726 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:20.252699    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n"
Jul 17 13:26:20.259067 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:20.259037    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n"]
Jul 17 13:26:20.334155 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:20.334125    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60e3f3b-8b0c-4c06-afad-0547507bbd38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n\" (UID: \"a60e3f3b-8b0c-4c06-afad-0547507bbd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n"
Jul 17 13:26:20.434663 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:20.434632    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60e3f3b-8b0c-4c06-afad-0547507bbd38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n\" (UID: \"a60e3f3b-8b0c-4c06-afad-0547507bbd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n"
Jul 17 13:26:20.435352 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:20.435329    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60e3f3b-8b0c-4c06-afad-0547507bbd38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n\" (UID: \"a60e3f3b-8b0c-4c06-afad-0547507bbd38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n"
Jul 17 13:26:20.565985 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:20.565921    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n"
Jul 17 13:26:20.703457 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:20.703430    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n"]
Jul 17 13:26:20.709039 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:26:20.709008    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60e3f3b_8b0c_4c06_afad_0547507bbd38.slice/crio-c72051864f72dddd3c6410ae7e6175efc79afbbcb9a54ef3dc3d38564ed8a0c2 WatchSource:0}: Error finding container c72051864f72dddd3c6410ae7e6175efc79afbbcb9a54ef3dc3d38564ed8a0c2: Status 404 returned error can't find the container with id c72051864f72dddd3c6410ae7e6175efc79afbbcb9a54ef3dc3d38564ed8a0c2
Jul 17 13:26:21.271715 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:21.271686    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n" event={"ID":"a60e3f3b-8b0c-4c06-afad-0547507bbd38","Type":"ContainerStarted","Data":"128391ee9247457024039da97741561cf40a282e9d72868af8db2e4ca219e435"}
Jul 17 13:26:21.272093 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:21.271736    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n" event={"ID":"a60e3f3b-8b0c-4c06-afad-0547507bbd38","Type":"ContainerStarted","Data":"c72051864f72dddd3c6410ae7e6175efc79afbbcb9a54ef3dc3d38564ed8a0c2"}
Jul 17 13:26:21.288411 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:21.288365    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n" podStartSLOduration=1.2883508 podStartE2EDuration="1.2883508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:26:21.287110494 +0000 UTC m=+3856.291157378" watchObservedRunningTime="2026-07-17 13:26:21.2883508 +0000 UTC m=+3856.292397682"
Jul 17 13:26:21.477297 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:21.477267    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc"]
Jul 17 13:26:21.480312 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:21.480296    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc"
Jul 17 13:26:21.487530 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:21.487507    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc"]
Jul 17 13:26:21.545404 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:21.545348    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5acf35-5896-4441-9f58-0b05ec506e57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc\" (UID: \"8f5acf35-5896-4441-9f58-0b05ec506e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc"
Jul 17 13:26:21.645864 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:21.645841    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5acf35-5896-4441-9f58-0b05ec506e57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc\" (UID: \"8f5acf35-5896-4441-9f58-0b05ec506e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc"
Jul 17 13:26:21.646412 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:21.646393    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5acf35-5896-4441-9f58-0b05ec506e57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc\" (UID: \"8f5acf35-5896-4441-9f58-0b05ec506e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc"
Jul 17 13:26:21.791207 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:21.791182    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc"
Jul 17 13:26:21.921393 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:21.921368    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc"]
Jul 17 13:26:21.923394 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:26:21.923367    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5acf35_5896_4441_9f58_0b05ec506e57.slice/crio-6f48753e4797748aa05dfd9043b87ecb57b75d0a35dbbad03313a066a47b4a98 WatchSource:0}: Error finding container 6f48753e4797748aa05dfd9043b87ecb57b75d0a35dbbad03313a066a47b4a98: Status 404 returned error can't find the container with id 6f48753e4797748aa05dfd9043b87ecb57b75d0a35dbbad03313a066a47b4a98
Jul 17 13:26:22.277898 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:22.277865    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc" event={"ID":"8f5acf35-5896-4441-9f58-0b05ec506e57","Type":"ContainerStarted","Data":"0b1ca82c8d6cbf0e97ae20c1f82a1b21b08671d632321bc718d402471bb5dcc4"}
Jul 17 13:26:22.278217 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:22.277927    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc" event={"ID":"8f5acf35-5896-4441-9f58-0b05ec506e57","Type":"ContainerStarted","Data":"6f48753e4797748aa05dfd9043b87ecb57b75d0a35dbbad03313a066a47b4a98"}
Jul 17 13:26:22.279272 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:22.279250    2569 generic.go:356] "Generic (PLEG): container finished" podID="a60e3f3b-8b0c-4c06-afad-0547507bbd38" containerID="128391ee9247457024039da97741561cf40a282e9d72868af8db2e4ca219e435" exitCode=6
Jul 17 13:26:22.279365 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:22.279288    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n" event={"ID":"a60e3f3b-8b0c-4c06-afad-0547507bbd38","Type":"ContainerDied","Data":"128391ee9247457024039da97741561cf40a282e9d72868af8db2e4ca219e435"}
Jul 17 13:26:22.279455 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:22.279444    2569 scope.go:117] "RemoveContainer" containerID="128391ee9247457024039da97741561cf40a282e9d72868af8db2e4ca219e435"
Jul 17 13:26:22.294804 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:22.294765    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc" podStartSLOduration=1.294735692 podStartE2EDuration="1.294735692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:26:22.292950033 +0000 UTC m=+3857.296996926" watchObservedRunningTime="2026-07-17 13:26:22.294735692 +0000 UTC m=+3857.298782565"
Jul 17 13:26:23.286209 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:23.286181    2569 generic.go:356] "Generic (PLEG): container finished" podID="8f5acf35-5896-4441-9f58-0b05ec506e57" containerID="0b1ca82c8d6cbf0e97ae20c1f82a1b21b08671d632321bc718d402471bb5dcc4" exitCode=6
Jul 17 13:26:23.286559 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:23.286264    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc" event={"ID":"8f5acf35-5896-4441-9f58-0b05ec506e57","Type":"ContainerDied","Data":"0b1ca82c8d6cbf0e97ae20c1f82a1b21b08671d632321bc718d402471bb5dcc4"}
Jul 17 13:26:23.286559 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:23.286417    2569 scope.go:117] "RemoveContainer" containerID="0b1ca82c8d6cbf0e97ae20c1f82a1b21b08671d632321bc718d402471bb5dcc4"
Jul 17 13:26:23.288194 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:23.288169    2569 generic.go:356] "Generic (PLEG): container finished" podID="a60e3f3b-8b0c-4c06-afad-0547507bbd38" containerID="adf2238706bc9f9c7866464b63d4fdc6d66d829eacd63f10c09da756698f4b91" exitCode=6
Jul 17 13:26:23.288306 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:23.288250    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n" event={"ID":"a60e3f3b-8b0c-4c06-afad-0547507bbd38","Type":"ContainerDied","Data":"adf2238706bc9f9c7866464b63d4fdc6d66d829eacd63f10c09da756698f4b91"}
Jul 17 13:26:23.288441 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:23.288303    2569 scope.go:117] "RemoveContainer" containerID="128391ee9247457024039da97741561cf40a282e9d72868af8db2e4ca219e435"
Jul 17 13:26:23.288493 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:23.288477    2569 scope.go:117] "RemoveContainer" containerID="adf2238706bc9f9c7866464b63d4fdc6d66d829eacd63f10c09da756698f4b91"
Jul 17 13:26:23.288684 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:26:23.288663    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n_odh-ai-gateway-infra(a60e3f3b-8b0c-4c06-afad-0547507bbd38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n" podUID="a60e3f3b-8b0c-4c06-afad-0547507bbd38"
Jul 17 13:26:23.870956 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:23.870931    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fa40d-9bdsp"]
Jul 17 13:26:23.874058 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:23.874042    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fa40d-9bdsp"
Jul 17 13:26:23.888595 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:23.888573    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fa40d-9bdsp"]
Jul 17 13:26:23.965570 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:23.965544    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c98eb2e-3e3f-4fd6-8701-8659bb1994cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6fa40d-9bdsp\" (UID: \"1c98eb2e-3e3f-4fd6-8701-8659bb1994cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fa40d-9bdsp"
Jul 17 13:26:24.066308 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:24.066288    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c98eb2e-3e3f-4fd6-8701-8659bb1994cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6fa40d-9bdsp\" (UID: \"1c98eb2e-3e3f-4fd6-8701-8659bb1994cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fa40d-9bdsp"
Jul 17 13:26:24.066840 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:24.066822    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c98eb2e-3e3f-4fd6-8701-8659bb1994cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6fa40d-9bdsp\" (UID: \"1c98eb2e-3e3f-4fd6-8701-8659bb1994cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fa40d-9bdsp"
Jul 17 13:26:24.193741 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:24.193690    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fa40d-9bdsp"
Jul 17 13:26:24.303408 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:24.303375    2569 generic.go:356] "Generic (PLEG): container finished" podID="8f5acf35-5896-4441-9f58-0b05ec506e57" containerID="828ee6813293c749c7a64acda41d428cabcbbbbb4b3f3c29a1c029d2bc121d95" exitCode=6
Jul 17 13:26:24.303837 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:24.303452    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc" event={"ID":"8f5acf35-5896-4441-9f58-0b05ec506e57","Type":"ContainerDied","Data":"828ee6813293c749c7a64acda41d428cabcbbbbb4b3f3c29a1c029d2bc121d95"}
Jul 17 13:26:24.303837 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:24.303556    2569 scope.go:117] "RemoveContainer" containerID="0b1ca82c8d6cbf0e97ae20c1f82a1b21b08671d632321bc718d402471bb5dcc4"
Jul 17 13:26:24.303837 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:24.303776    2569 scope.go:117] "RemoveContainer" containerID="828ee6813293c749c7a64acda41d428cabcbbbbb4b3f3c29a1c029d2bc121d95"
Jul 17 13:26:24.304099 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:26:24.304037    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc_odh-ai-gateway-infra(8f5acf35-5896-4441-9f58-0b05ec506e57)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc" podUID="8f5acf35-5896-4441-9f58-0b05ec506e57"
Jul 17 13:26:24.305645 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:24.305624    2569 scope.go:117] "RemoveContainer" containerID="adf2238706bc9f9c7866464b63d4fdc6d66d829eacd63f10c09da756698f4b91"
Jul 17 13:26:24.305902 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:26:24.305863    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n_odh-ai-gateway-infra(a60e3f3b-8b0c-4c06-afad-0547507bbd38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n" podUID="a60e3f3b-8b0c-4c06-afad-0547507bbd38"
Jul 17 13:26:24.331521 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:24.331501    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fa40d-9bdsp"]
Jul 17 13:26:24.333570 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:26:24.333548    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c98eb2e_3e3f_4fd6_8701_8659bb1994cd.slice/crio-d6b28409bf65be73e345dfa47a9ff6a18f71db479f33519c270e977500907b80 WatchSource:0}: Error finding container d6b28409bf65be73e345dfa47a9ff6a18f71db479f33519c270e977500907b80: Status 404 returned error can't find the container with id d6b28409bf65be73e345dfa47a9ff6a18f71db479f33519c270e977500907b80
Jul 17 13:26:24.628340 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:24.628319    2569 scope.go:117] "RemoveContainer" containerID="3408c25eaceae7a34a7c98f0e522ac0abf9592d192ec8c8afa6ef20124ecca73"
Jul 17 13:26:25.311679 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:25.311651    2569 generic.go:356] "Generic (PLEG): container finished" podID="1c98eb2e-3e3f-4fd6-8701-8659bb1994cd" containerID="e13614b07751c372d84d1fb51af3cbae853914eb721478529d060ead9e8bd373" exitCode=0
Jul 17 13:26:25.312054 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:25.311736    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fa40d-9bdsp" event={"ID":"1c98eb2e-3e3f-4fd6-8701-8659bb1994cd","Type":"ContainerDied","Data":"e13614b07751c372d84d1fb51af3cbae853914eb721478529d060ead9e8bd373"}
Jul 17 13:26:25.312054 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:25.311831    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fa40d-9bdsp" event={"ID":"1c98eb2e-3e3f-4fd6-8701-8659bb1994cd","Type":"ContainerStarted","Data":"d6b28409bf65be73e345dfa47a9ff6a18f71db479f33519c270e977500907b80"}
Jul 17 13:26:25.313790 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:25.313770    2569 generic.go:356] "Generic (PLEG): container finished" podID="86e3408e-a2d2-482c-837e-5dc77aa3c6b4" containerID="8983e292e7dd094cdbd3ec423935dfe776b31d04941c3244fbbcbf8c0f51d6a1" exitCode=6
Jul 17 13:26:25.313895 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:25.313835    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr" event={"ID":"86e3408e-a2d2-482c-837e-5dc77aa3c6b4","Type":"ContainerDied","Data":"8983e292e7dd094cdbd3ec423935dfe776b31d04941c3244fbbcbf8c0f51d6a1"}
Jul 17 13:26:25.313895 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:25.313886    2569 scope.go:117] "RemoveContainer" containerID="3408c25eaceae7a34a7c98f0e522ac0abf9592d192ec8c8afa6ef20124ecca73"
Jul 17 13:26:25.314146 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:25.314129    2569 scope.go:117] "RemoveContainer" containerID="8983e292e7dd094cdbd3ec423935dfe776b31d04941c3244fbbcbf8c0f51d6a1"
Jul 17 13:26:25.314352 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:26:25.314326    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr_odh-ai-gateway-infra(86e3408e-a2d2-482c-837e-5dc77aa3c6b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr" podUID="86e3408e-a2d2-482c-837e-5dc77aa3c6b4"
Jul 17 13:26:25.315592 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:25.315577    2569 scope.go:117] "RemoveContainer" containerID="828ee6813293c749c7a64acda41d428cabcbbbbb4b3f3c29a1c029d2bc121d95"
Jul 17 13:26:25.315735 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:26:25.315721    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc_odh-ai-gateway-infra(8f5acf35-5896-4441-9f58-0b05ec506e57)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc" podUID="8f5acf35-5896-4441-9f58-0b05ec506e57"
Jul 17 13:26:26.351231 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:26.351198    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr"]
Jul 17 13:26:26.477917 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:26.477883    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fa40d-9bdsp"
Jul 17 13:26:26.511599 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:26.511579    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr"
Jul 17 13:26:26.586106 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:26.586039    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c98eb2e-3e3f-4fd6-8701-8659bb1994cd-maas-api-service-ca\") pod \"1c98eb2e-3e3f-4fd6-8701-8659bb1994cd\" (UID: \"1c98eb2e-3e3f-4fd6-8701-8659bb1994cd\") "
Jul 17 13:26:26.586222 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:26.586121    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86e3408e-a2d2-482c-837e-5dc77aa3c6b4-maas-api-service-ca\") pod \"86e3408e-a2d2-482c-837e-5dc77aa3c6b4\" (UID: \"86e3408e-a2d2-482c-837e-5dc77aa3c6b4\") "
Jul 17 13:26:26.586393 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:26.586335    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c98eb2e-3e3f-4fd6-8701-8659bb1994cd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c98eb2e-3e3f-4fd6-8701-8659bb1994cd" (UID: "1c98eb2e-3e3f-4fd6-8701-8659bb1994cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:26:26.586458 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:26.586442    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86e3408e-a2d2-482c-837e-5dc77aa3c6b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86e3408e-a2d2-482c-837e-5dc77aa3c6b4" (UID: "86e3408e-a2d2-482c-837e-5dc77aa3c6b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:26:26.686801 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:26.686779    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86e3408e-a2d2-482c-837e-5dc77aa3c6b4-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:26:26.686801 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:26.686803    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c98eb2e-3e3f-4fd6-8701-8659bb1994cd-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:26:27.328540 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:27.328518    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fa40d-9bdsp"
Jul 17 13:26:27.328716 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:27.328541    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6fa40d-9bdsp" event={"ID":"1c98eb2e-3e3f-4fd6-8701-8659bb1994cd","Type":"ContainerDied","Data":"d6b28409bf65be73e345dfa47a9ff6a18f71db479f33519c270e977500907b80"}
Jul 17 13:26:27.328716 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:27.328594    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6b28409bf65be73e345dfa47a9ff6a18f71db479f33519c270e977500907b80"
Jul 17 13:26:27.329977 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:27.329958    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr"
Jul 17 13:26:27.329977 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:27.329963    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr" event={"ID":"86e3408e-a2d2-482c-837e-5dc77aa3c6b4","Type":"ContainerDied","Data":"65b7a383c11cf3e86a1925f71006f989fae09663708de351883d11434bb11f85"}
Jul 17 13:26:27.330158 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:27.330016    2569 scope.go:117] "RemoveContainer" containerID="8983e292e7dd094cdbd3ec423935dfe776b31d04941c3244fbbcbf8c0f51d6a1"
Jul 17 13:26:27.355393 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:27.355369    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr"]
Jul 17 13:26:27.359227 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:27.359197    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-8hpcr"]
Jul 17 13:26:27.634477 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:27.634427    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86e3408e-a2d2-482c-837e-5dc77aa3c6b4" path="/var/lib/kubelet/pods/86e3408e-a2d2-482c-837e-5dc77aa3c6b4/volumes"
Jul 17 13:26:34.369147 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:34.369112    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"]
Jul 17 13:26:34.369594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:34.369355    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59" podUID="fd91ba59-2a5a-48c0-a791-1def0a6762b8" containerName="maas-api" containerID="cri-o://310cf33dec08593c81cd886a4c996a7770490e35fbb4c5ecc2a8d926d43c9297" gracePeriod=30
Jul 17 13:26:34.616933 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:34.616908    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"
Jul 17 13:26:34.747641 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:34.747616    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd91ba59-2a5a-48c0-a791-1def0a6762b8-maas-api-tls\") pod \"fd91ba59-2a5a-48c0-a791-1def0a6762b8\" (UID: \"fd91ba59-2a5a-48c0-a791-1def0a6762b8\") "
Jul 17 13:26:34.747792 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:34.747646    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd91ba59-2a5a-48c0-a791-1def0a6762b8-openshift-service-ca\") pod \"fd91ba59-2a5a-48c0-a791-1def0a6762b8\" (UID: \"fd91ba59-2a5a-48c0-a791-1def0a6762b8\") "
Jul 17 13:26:34.747792 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:34.747670    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rsr8\" (UniqueName: \"kubernetes.io/projected/fd91ba59-2a5a-48c0-a791-1def0a6762b8-kube-api-access-8rsr8\") pod \"fd91ba59-2a5a-48c0-a791-1def0a6762b8\" (UID: \"fd91ba59-2a5a-48c0-a791-1def0a6762b8\") "
Jul 17 13:26:34.748087 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:34.748066    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd91ba59-2a5a-48c0-a791-1def0a6762b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fd91ba59-2a5a-48c0-a791-1def0a6762b8" (UID: "fd91ba59-2a5a-48c0-a791-1def0a6762b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:26:34.749764 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:34.749733    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd91ba59-2a5a-48c0-a791-1def0a6762b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fd91ba59-2a5a-48c0-a791-1def0a6762b8" (UID: "fd91ba59-2a5a-48c0-a791-1def0a6762b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:26:34.749859 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:34.749840    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd91ba59-2a5a-48c0-a791-1def0a6762b8-kube-api-access-8rsr8" (OuterVolumeSpecName: "kube-api-access-8rsr8") pod "fd91ba59-2a5a-48c0-a791-1def0a6762b8" (UID: "fd91ba59-2a5a-48c0-a791-1def0a6762b8"). InnerVolumeSpecName "kube-api-access-8rsr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:26:34.848545 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:34.848524    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd91ba59-2a5a-48c0-a791-1def0a6762b8-maas-api-tls\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:26:34.848545 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:34.848544    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd91ba59-2a5a-48c0-a791-1def0a6762b8-openshift-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:26:34.848680 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:34.848556    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rsr8\" (UniqueName: \"kubernetes.io/projected/fd91ba59-2a5a-48c0-a791-1def0a6762b8-kube-api-access-8rsr8\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:26:35.386039 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:35.386000    2569 generic.go:356] "Generic (PLEG): container finished" podID="fd91ba59-2a5a-48c0-a791-1def0a6762b8" containerID="310cf33dec08593c81cd886a4c996a7770490e35fbb4c5ecc2a8d926d43c9297" exitCode=0
Jul 17 13:26:35.386435 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:35.386064    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"
Jul 17 13:26:35.386435 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:35.386077    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59" event={"ID":"fd91ba59-2a5a-48c0-a791-1def0a6762b8","Type":"ContainerDied","Data":"310cf33dec08593c81cd886a4c996a7770490e35fbb4c5ecc2a8d926d43c9297"}
Jul 17 13:26:35.386435 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:35.386155    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59" event={"ID":"fd91ba59-2a5a-48c0-a791-1def0a6762b8","Type":"ContainerDied","Data":"8d2536f7d7ed2f136a15c91c46c03a579c34264c4ac02ea9e904e35b47c5a8a8"}
Jul 17 13:26:35.386435 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:35.386178    2569 scope.go:117] "RemoveContainer" containerID="310cf33dec08593c81cd886a4c996a7770490e35fbb4c5ecc2a8d926d43c9297"
Jul 17 13:26:35.395647 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:35.395623    2569 scope.go:117] "RemoveContainer" containerID="310cf33dec08593c81cd886a4c996a7770490e35fbb4c5ecc2a8d926d43c9297"
Jul 17 13:26:35.395982 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:26:35.395961    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"310cf33dec08593c81cd886a4c996a7770490e35fbb4c5ecc2a8d926d43c9297\": container with ID starting with 310cf33dec08593c81cd886a4c996a7770490e35fbb4c5ecc2a8d926d43c9297 not found: ID does not exist" containerID="310cf33dec08593c81cd886a4c996a7770490e35fbb4c5ecc2a8d926d43c9297"
Jul 17 13:26:35.396037 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:35.395990    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310cf33dec08593c81cd886a4c996a7770490e35fbb4c5ecc2a8d926d43c9297"} err="failed to get container status \"310cf33dec08593c81cd886a4c996a7770490e35fbb4c5ecc2a8d926d43c9297\": rpc error: code = NotFound desc = could not find container \"310cf33dec08593c81cd886a4c996a7770490e35fbb4c5ecc2a8d926d43c9297\": container with ID starting with 310cf33dec08593c81cd886a4c996a7770490e35fbb4c5ecc2a8d926d43c9297 not found: ID does not exist"
Jul 17 13:26:35.408159 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:35.408138    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"]
Jul 17 13:26:35.416106 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:35.416085    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6fa40d-7f544cc4cd-x4n59"]
Jul 17 13:26:35.631147 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:35.631127    2569 scope.go:117] "RemoveContainer" containerID="adf2238706bc9f9c7866464b63d4fdc6d66d829eacd63f10c09da756698f4b91"
Jul 17 13:26:35.637019 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:35.636968    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd91ba59-2a5a-48c0-a791-1def0a6762b8" path="/var/lib/kubelet/pods/fd91ba59-2a5a-48c0-a791-1def0a6762b8/volumes"
Jul 17 13:26:36.392010 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:36.391985    2569 generic.go:356] "Generic (PLEG): container finished" podID="a60e3f3b-8b0c-4c06-afad-0547507bbd38" containerID="7490fa459a9055345292c01c0fd2ceee937ce8615e726b22994169a3754f5598" exitCode=6
Jul 17 13:26:36.392329 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:36.392057    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n" event={"ID":"a60e3f3b-8b0c-4c06-afad-0547507bbd38","Type":"ContainerDied","Data":"7490fa459a9055345292c01c0fd2ceee937ce8615e726b22994169a3754f5598"}
Jul 17 13:26:36.392329 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:36.392107    2569 scope.go:117] "RemoveContainer" containerID="adf2238706bc9f9c7866464b63d4fdc6d66d829eacd63f10c09da756698f4b91"
Jul 17 13:26:36.392441 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:36.392380    2569 scope.go:117] "RemoveContainer" containerID="7490fa459a9055345292c01c0fd2ceee937ce8615e726b22994169a3754f5598"
Jul 17 13:26:36.392571 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:26:36.392551    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n_odh-ai-gateway-infra(a60e3f3b-8b0c-4c06-afad-0547507bbd38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n" podUID="a60e3f3b-8b0c-4c06-afad-0547507bbd38"
Jul 17 13:26:37.421100 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:37.421066    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n"]
Jul 17 13:26:37.554898 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:37.554876    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n"
Jul 17 13:26:37.666538 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:37.666513    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60e3f3b-8b0c-4c06-afad-0547507bbd38-maas-api-service-ca\") pod \"a60e3f3b-8b0c-4c06-afad-0547507bbd38\" (UID: \"a60e3f3b-8b0c-4c06-afad-0547507bbd38\") "
Jul 17 13:26:37.666858 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:37.666838    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60e3f3b-8b0c-4c06-afad-0547507bbd38-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a60e3f3b-8b0c-4c06-afad-0547507bbd38" (UID: "a60e3f3b-8b0c-4c06-afad-0547507bbd38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:26:37.670040 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:37.670019    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:26:37.767613 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:37.767584    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60e3f3b-8b0c-4c06-afad-0547507bbd38-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:26:38.407042 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:38.407011    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n" event={"ID":"a60e3f3b-8b0c-4c06-afad-0547507bbd38","Type":"ContainerDied","Data":"c72051864f72dddd3c6410ae7e6175efc79afbbcb9a54ef3dc3d38564ed8a0c2"}
Jul 17 13:26:38.407184 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:38.407048    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n"
Jul 17 13:26:38.407184 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:38.407084    2569 scope.go:117] "RemoveContainer" containerID="7490fa459a9055345292c01c0fd2ceee937ce8615e726b22994169a3754f5598"
Jul 17 13:26:38.428276 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:38.428251    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n"]
Jul 17 13:26:38.439200 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:38.439175    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-8ht2n"]
Jul 17 13:26:38.628210 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:38.628189    2569 scope.go:117] "RemoveContainer" containerID="828ee6813293c749c7a64acda41d428cabcbbbbb4b3f3c29a1c029d2bc121d95"
Jul 17 13:26:39.414304 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:39.414276    2569 generic.go:356] "Generic (PLEG): container finished" podID="8f5acf35-5896-4441-9f58-0b05ec506e57" containerID="fb21d70e1fb3390566b0785451386f2fbebcedf6109280694e29e67a429488a8" exitCode=6
Jul 17 13:26:39.414415 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:39.414346    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc" event={"ID":"8f5acf35-5896-4441-9f58-0b05ec506e57","Type":"ContainerDied","Data":"fb21d70e1fb3390566b0785451386f2fbebcedf6109280694e29e67a429488a8"}
Jul 17 13:26:39.414459 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:39.414428    2569 scope.go:117] "RemoveContainer" containerID="828ee6813293c749c7a64acda41d428cabcbbbbb4b3f3c29a1c029d2bc121d95"
Jul 17 13:26:39.414668 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:39.414653    2569 scope.go:117] "RemoveContainer" containerID="fb21d70e1fb3390566b0785451386f2fbebcedf6109280694e29e67a429488a8"
Jul 17 13:26:39.414869 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:26:39.414850    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc_odh-ai-gateway-infra(8f5acf35-5896-4441-9f58-0b05ec506e57)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc" podUID="8f5acf35-5896-4441-9f58-0b05ec506e57"
Jul 17 13:26:39.633673 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:39.633619    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a60e3f3b-8b0c-4c06-afad-0547507bbd38" path="/var/lib/kubelet/pods/a60e3f3b-8b0c-4c06-afad-0547507bbd38/volumes"
Jul 17 13:26:40.445507 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:40.445474    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc"]
Jul 17 13:26:40.579272 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:40.579248    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc"
Jul 17 13:26:40.691769 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:40.691728    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5acf35-5896-4441-9f58-0b05ec506e57-maas-api-service-ca\") pod \"8f5acf35-5896-4441-9f58-0b05ec506e57\" (UID: \"8f5acf35-5896-4441-9f58-0b05ec506e57\") "
Jul 17 13:26:40.692100 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:40.692067    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f5acf35-5896-4441-9f58-0b05ec506e57-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f5acf35-5896-4441-9f58-0b05ec506e57" (UID: "8f5acf35-5896-4441-9f58-0b05ec506e57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:26:40.793171 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:40.793134    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f5acf35-5896-4441-9f58-0b05ec506e57-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:26:41.428227 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:41.428203    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc"
Jul 17 13:26:41.428389 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:41.428237    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc" event={"ID":"8f5acf35-5896-4441-9f58-0b05ec506e57","Type":"ContainerDied","Data":"6f48753e4797748aa05dfd9043b87ecb57b75d0a35dbbad03313a066a47b4a98"}
Jul 17 13:26:41.428389 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:41.428305    2569 scope.go:117] "RemoveContainer" containerID="fb21d70e1fb3390566b0785451386f2fbebcedf6109280694e29e67a429488a8"
Jul 17 13:26:41.451519 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:41.451492    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc"]
Jul 17 13:26:41.457855 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:41.457832    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-xrdjc"]
Jul 17 13:26:41.634132 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:41.634099    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f5acf35-5896-4441-9f58-0b05ec506e57" path="/var/lib/kubelet/pods/8f5acf35-5896-4441-9f58-0b05ec506e57/volumes"
Jul 17 13:26:53.612125 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612088    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"]
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612420    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c98eb2e-3e3f-4fd6-8701-8659bb1994cd" containerName="revoke-keys"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612431    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c98eb2e-3e3f-4fd6-8701-8659bb1994cd" containerName="revoke-keys"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612446    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60e3f3b-8b0c-4c06-afad-0547507bbd38" containerName="revoke-keys"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612452    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60e3f3b-8b0c-4c06-afad-0547507bbd38" containerName="revoke-keys"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612460    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd91ba59-2a5a-48c0-a791-1def0a6762b8" containerName="maas-api"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612466    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd91ba59-2a5a-48c0-a791-1def0a6762b8" containerName="maas-api"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612475    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f5acf35-5896-4441-9f58-0b05ec506e57" containerName="revoke-keys"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612480    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f5acf35-5896-4441-9f58-0b05ec506e57" containerName="revoke-keys"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612486    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86e3408e-a2d2-482c-837e-5dc77aa3c6b4" containerName="revoke-keys"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612491    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="86e3408e-a2d2-482c-837e-5dc77aa3c6b4" containerName="revoke-keys"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612502    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86e3408e-a2d2-482c-837e-5dc77aa3c6b4" containerName="revoke-keys"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612507    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="86e3408e-a2d2-482c-837e-5dc77aa3c6b4" containerName="revoke-keys"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612513    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f5acf35-5896-4441-9f58-0b05ec506e57" containerName="revoke-keys"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612517    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f5acf35-5896-4441-9f58-0b05ec506e57" containerName="revoke-keys"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612523    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f5acf35-5896-4441-9f58-0b05ec506e57" containerName="revoke-keys"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612528    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f5acf35-5896-4441-9f58-0b05ec506e57" containerName="revoke-keys"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612535    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86e3408e-a2d2-482c-837e-5dc77aa3c6b4" containerName="revoke-keys"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612539    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="86e3408e-a2d2-482c-837e-5dc77aa3c6b4" containerName="revoke-keys"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612546    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60e3f3b-8b0c-4c06-afad-0547507bbd38" containerName="revoke-keys"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612551    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60e3f3b-8b0c-4c06-afad-0547507bbd38" containerName="revoke-keys"
Jul 17 13:26:53.612594 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612595    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f5acf35-5896-4441-9f58-0b05ec506e57" containerName="revoke-keys"
Jul 17 13:26:53.613707 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612601    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="86e3408e-a2d2-482c-837e-5dc77aa3c6b4" containerName="revoke-keys"
Jul 17 13:26:53.613707 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612629    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60e3f3b-8b0c-4c06-afad-0547507bbd38" containerName="revoke-keys"
Jul 17 13:26:53.613707 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612636    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60e3f3b-8b0c-4c06-afad-0547507bbd38" containerName="revoke-keys"
Jul 17 13:26:53.613707 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612643    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f5acf35-5896-4441-9f58-0b05ec506e57" containerName="revoke-keys"
Jul 17 13:26:53.613707 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612648    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f5acf35-5896-4441-9f58-0b05ec506e57" containerName="revoke-keys"
Jul 17 13:26:53.613707 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612654    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60e3f3b-8b0c-4c06-afad-0547507bbd38" containerName="revoke-keys"
Jul 17 13:26:53.613707 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612662    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd91ba59-2a5a-48c0-a791-1def0a6762b8" containerName="maas-api"
Jul 17 13:26:53.613707 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612666    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c98eb2e-3e3f-4fd6-8701-8659bb1994cd" containerName="revoke-keys"
Jul 17 13:26:53.613707 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.612671    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="86e3408e-a2d2-482c-837e-5dc77aa3c6b4" containerName="revoke-keys"
Jul 17 13:26:53.615653 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.615635    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"
Jul 17 13:26:53.622836 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.622810    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-e92eab-serving-cert\""
Jul 17 13:26:53.622962 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.622876    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:26:53.623932 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.623916    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 13:26:53.624024 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.623922    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lwvzj\""
Jul 17 13:26:53.638724 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.638702    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"]
Jul 17 13:26:53.690299 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.690275    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkcgn\" (UniqueName: \"kubernetes.io/projected/0231dd62-5202-4852-8853-324f8b8d94c0-kube-api-access-dkcgn\") pod \"maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt\" (UID: \"0231dd62-5202-4852-8853-324f8b8d94c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"
Jul 17 13:26:53.690414 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.690391    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0231dd62-5202-4852-8853-324f8b8d94c0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt\" (UID: \"0231dd62-5202-4852-8853-324f8b8d94c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"
Jul 17 13:26:53.690462 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.690419    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0231dd62-5202-4852-8853-324f8b8d94c0-maas-api-tls\") pod \"maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt\" (UID: \"0231dd62-5202-4852-8853-324f8b8d94c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"
Jul 17 13:26:53.791662 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.791625    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0231dd62-5202-4852-8853-324f8b8d94c0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt\" (UID: \"0231dd62-5202-4852-8853-324f8b8d94c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"
Jul 17 13:26:53.791837 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.791665    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0231dd62-5202-4852-8853-324f8b8d94c0-maas-api-tls\") pod \"maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt\" (UID: \"0231dd62-5202-4852-8853-324f8b8d94c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"
Jul 17 13:26:53.791837 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.791727    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkcgn\" (UniqueName: \"kubernetes.io/projected/0231dd62-5202-4852-8853-324f8b8d94c0-kube-api-access-dkcgn\") pod \"maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt\" (UID: \"0231dd62-5202-4852-8853-324f8b8d94c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"
Jul 17 13:26:53.792239 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.792217    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0231dd62-5202-4852-8853-324f8b8d94c0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt\" (UID: \"0231dd62-5202-4852-8853-324f8b8d94c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"
Jul 17 13:26:53.794049 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.794026    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0231dd62-5202-4852-8853-324f8b8d94c0-maas-api-tls\") pod \"maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt\" (UID: \"0231dd62-5202-4852-8853-324f8b8d94c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"
Jul 17 13:26:53.800460 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.800441    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkcgn\" (UniqueName: \"kubernetes.io/projected/0231dd62-5202-4852-8853-324f8b8d94c0-kube-api-access-dkcgn\") pod \"maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt\" (UID: \"0231dd62-5202-4852-8853-324f8b8d94c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"
Jul 17 13:26:53.926010 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:53.925942    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"
Jul 17 13:26:54.051560 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:54.051531    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"]
Jul 17 13:26:54.053679 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:26:54.053645    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0231dd62_5202_4852_8853_324f8b8d94c0.slice/crio-827fc19fc3508e00e4f8a1b6420166c4e00e3da312aa9fd66f4bd0ece3a3b40c WatchSource:0}: Error finding container 827fc19fc3508e00e4f8a1b6420166c4e00e3da312aa9fd66f4bd0ece3a3b40c: Status 404 returned error can't find the container with id 827fc19fc3508e00e4f8a1b6420166c4e00e3da312aa9fd66f4bd0ece3a3b40c
Jul 17 13:26:54.498000 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:54.497973    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt" event={"ID":"0231dd62-5202-4852-8853-324f8b8d94c0","Type":"ContainerStarted","Data":"016aadb2971e520934d763e221c4d78e91f213b51cd620ee6e28c0fca3a6caf0"}
Jul 17 13:26:54.498103 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:54.498024    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt" event={"ID":"0231dd62-5202-4852-8853-324f8b8d94c0","Type":"ContainerStarted","Data":"827fc19fc3508e00e4f8a1b6420166c4e00e3da312aa9fd66f4bd0ece3a3b40c"}
Jul 17 13:26:54.498103 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:54.498056    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"
Jul 17 13:26:54.518387 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:54.518342    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt" podStartSLOduration=1.14706664 podStartE2EDuration="1.518328153s" totalImagesPullingTime="371.261513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:26:54.517619624 +0000 UTC m=+3889.521666510" watchObservedRunningTime="2026-07-17 13:26:54.518328153 +0000 UTC m=+3889.522375034"
Jul 17 13:26:59.009504 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:59.009475    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx"]
Jul 17 13:26:59.009882 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:59.009806    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60e3f3b-8b0c-4c06-afad-0547507bbd38" containerName="revoke-keys"
Jul 17 13:26:59.009882 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:59.009818    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60e3f3b-8b0c-4c06-afad-0547507bbd38" containerName="revoke-keys"
Jul 17 13:26:59.009882 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:59.009877    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="86e3408e-a2d2-482c-837e-5dc77aa3c6b4" containerName="revoke-keys"
Jul 17 13:26:59.012953 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:59.012932    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx"
Jul 17 13:26:59.015654 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:59.015637    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:26:59.022674 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:59.022645    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx"]
Jul 17 13:26:59.033415 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:59.033389    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72e2017e-92fe-4a8b-bd19-13da2e0f2407-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx\" (UID: \"72e2017e-92fe-4a8b-bd19-13da2e0f2407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx"
Jul 17 13:26:59.134101 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:59.134081    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72e2017e-92fe-4a8b-bd19-13da2e0f2407-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx\" (UID: \"72e2017e-92fe-4a8b-bd19-13da2e0f2407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx"
Jul 17 13:26:59.134616 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:59.134598    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72e2017e-92fe-4a8b-bd19-13da2e0f2407-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx\" (UID: \"72e2017e-92fe-4a8b-bd19-13da2e0f2407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx"
Jul 17 13:26:59.272732 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:59.272674    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:26:59.325124 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:59.325103    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx"
Jul 17 13:26:59.458472 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:59.458449    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx"]
Jul 17 13:26:59.460937 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:26:59.460904    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e2017e_92fe_4a8b_bd19_13da2e0f2407.slice/crio-892f5ca0797a38742a772363468977f1455c53729bb821da92c927cf523ebe29 WatchSource:0}: Error finding container 892f5ca0797a38742a772363468977f1455c53729bb821da92c927cf523ebe29: Status 404 returned error can't find the container with id 892f5ca0797a38742a772363468977f1455c53729bb821da92c927cf523ebe29
Jul 17 13:26:59.529650 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:26:59.529595    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx" event={"ID":"72e2017e-92fe-4a8b-bd19-13da2e0f2407","Type":"ContainerStarted","Data":"892f5ca0797a38742a772363468977f1455c53729bb821da92c927cf523ebe29"}
Jul 17 13:27:00.508745 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:00.508721    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"
Jul 17 13:27:00.536611 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:00.536583    2569 generic.go:356] "Generic (PLEG): container finished" podID="72e2017e-92fe-4a8b-bd19-13da2e0f2407" containerID="fbacbdea1c656911466f09a77b9c3d4ee9fb617032c9203674e15884f8e90a61" exitCode=6
Jul 17 13:27:00.536767 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:00.536620    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx" event={"ID":"72e2017e-92fe-4a8b-bd19-13da2e0f2407","Type":"ContainerDied","Data":"fbacbdea1c656911466f09a77b9c3d4ee9fb617032c9203674e15884f8e90a61"}
Jul 17 13:27:00.536883 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:00.536866    2569 scope.go:117] "RemoveContainer" containerID="fbacbdea1c656911466f09a77b9c3d4ee9fb617032c9203674e15884f8e90a61"
Jul 17 13:27:01.544265 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:01.544235    2569 generic.go:356] "Generic (PLEG): container finished" podID="72e2017e-92fe-4a8b-bd19-13da2e0f2407" containerID="f02287275f3369940302cad3a18b5fde0602a7a884cd846cf65fe4900307e8c1" exitCode=6
Jul 17 13:27:01.544648 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:01.544320    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx" event={"ID":"72e2017e-92fe-4a8b-bd19-13da2e0f2407","Type":"ContainerDied","Data":"f02287275f3369940302cad3a18b5fde0602a7a884cd846cf65fe4900307e8c1"}
Jul 17 13:27:01.544648 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:01.544380    2569 scope.go:117] "RemoveContainer" containerID="fbacbdea1c656911466f09a77b9c3d4ee9fb617032c9203674e15884f8e90a61"
Jul 17 13:27:01.544648 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:01.544484    2569 scope.go:117] "RemoveContainer" containerID="f02287275f3369940302cad3a18b5fde0602a7a884cd846cf65fe4900307e8c1"
Jul 17 13:27:01.544810 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:27:01.544698    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx_odh-ai-gateway-infra(72e2017e-92fe-4a8b-bd19-13da2e0f2407)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx" podUID="72e2017e-92fe-4a8b-bd19-13da2e0f2407"
Jul 17 13:27:01.660889 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:01.660864    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"]
Jul 17 13:27:01.664513 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:01.664494    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"
Jul 17 13:27:01.668160 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:01.668131    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-5c3765-serving-cert\""
Jul 17 13:27:01.682137 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:01.682116    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"]
Jul 17 13:27:01.753408 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:01.753380    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-maas-api-tls\") pod \"maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq\" (UID: \"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"
Jul 17 13:27:01.753500 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:01.753420    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq\" (UID: \"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"
Jul 17 13:27:01.753500 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:01.753484    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg4vp\" (UniqueName: \"kubernetes.io/projected/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-kube-api-access-xg4vp\") pod \"maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq\" (UID: \"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"
Jul 17 13:27:01.854641 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:01.854575    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-maas-api-tls\") pod \"maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq\" (UID: \"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"
Jul 17 13:27:01.854641 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:01.854606    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq\" (UID: \"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"
Jul 17 13:27:01.854641 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:01.854636    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg4vp\" (UniqueName: \"kubernetes.io/projected/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-kube-api-access-xg4vp\") pod \"maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq\" (UID: \"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"
Jul 17 13:27:01.854864 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:27:01.854762    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-serving-cert: secret "maas-api-e2e-shared-b-5c3765-serving-cert" not found
Jul 17 13:27:01.854864 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:27:01.854834    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-maas-api-tls podName:57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3 nodeName:}" failed. No retries permitted until 2026-07-17 13:27:02.35481388 +0000 UTC m=+3897.358860755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-maas-api-tls") pod "maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq" (UID: "57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3") : secret "maas-api-e2e-shared-b-5c3765-serving-cert" not found
Jul 17 13:27:01.855277 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:01.855260    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq\" (UID: \"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"
Jul 17 13:27:01.891321 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:01.891293    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg4vp\" (UniqueName: \"kubernetes.io/projected/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-kube-api-access-xg4vp\") pod \"maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq\" (UID: \"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"
Jul 17 13:27:02.358385 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:02.358357    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-maas-api-tls\") pod \"maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq\" (UID: \"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"
Jul 17 13:27:02.360633 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:02.360610    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-maas-api-tls\") pod \"maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq\" (UID: \"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"
Jul 17 13:27:02.558843 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:02.558818    2569 scope.go:117] "RemoveContainer" containerID="f02287275f3369940302cad3a18b5fde0602a7a884cd846cf65fe4900307e8c1"
Jul 17 13:27:02.559286 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:27:02.559020    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx_odh-ai-gateway-infra(72e2017e-92fe-4a8b-bd19-13da2e0f2407)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx" podUID="72e2017e-92fe-4a8b-bd19-13da2e0f2407"
Jul 17 13:27:02.575397 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:02.575377    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"
Jul 17 13:27:02.718033 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:02.718010    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"]
Jul 17 13:27:02.719457 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:27:02.719431    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57eeaded_3b80_4b86_afa5_fa0fb8f7f0c3.slice/crio-e88a67bf12108072695240a6fd245698ef839585b819263c6574f9f5ccbc7063 WatchSource:0}: Error finding container e88a67bf12108072695240a6fd245698ef839585b819263c6574f9f5ccbc7063: Status 404 returned error can't find the container with id e88a67bf12108072695240a6fd245698ef839585b819263c6574f9f5ccbc7063
Jul 17 13:27:03.564865 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:03.564779    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq" event={"ID":"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3","Type":"ContainerStarted","Data":"b8a306a676bd246e164cce1169d935247fa3bb56b195b4f9445e2f1f7c085249"}
Jul 17 13:27:03.564865 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:03.564839    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq" event={"ID":"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3","Type":"ContainerStarted","Data":"e88a67bf12108072695240a6fd245698ef839585b819263c6574f9f5ccbc7063"}
Jul 17 13:27:03.564865 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:03.564867    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"
Jul 17 13:27:03.585007 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:03.584961    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq" podStartSLOduration=2.039025893 podStartE2EDuration="2.584948102s" totalImagesPullingTime="545.922209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:27:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:27:03.583129595 +0000 UTC m=+3898.587176481" watchObservedRunningTime="2026-07-17 13:27:03.584948102 +0000 UTC m=+3898.588994983"
Jul 17 13:27:09.575984 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:09.575946    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"
Jul 17 13:27:11.528027 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:11.527984    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6"]
Jul 17 13:27:11.532933 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:11.532912    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6"
Jul 17 13:27:11.539502 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:11.539468    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6"]
Jul 17 13:27:11.626351 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:11.626323    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ff745d-6aa9-441b-b52a-d50de67bda38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6\" (UID: \"48ff745d-6aa9-441b-b52a-d50de67bda38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6"
Jul 17 13:27:11.726741 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:11.726703    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ff745d-6aa9-441b-b52a-d50de67bda38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6\" (UID: \"48ff745d-6aa9-441b-b52a-d50de67bda38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6"
Jul 17 13:27:11.727274 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:11.727251    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ff745d-6aa9-441b-b52a-d50de67bda38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6\" (UID: \"48ff745d-6aa9-441b-b52a-d50de67bda38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6"
Jul 17 13:27:11.845146 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:11.845088    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6"
Jul 17 13:27:11.972620 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:11.972589    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:27:12.179218 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:12.179156    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6"]
Jul 17 13:27:12.182148 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:27:12.182122    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ff745d_6aa9_441b_b52a_d50de67bda38.slice/crio-16ac41cc40c6ccde8cbc41abaf646f2cdc1ed594fc2301555873601e681f6fb6 WatchSource:0}: Error finding container 16ac41cc40c6ccde8cbc41abaf646f2cdc1ed594fc2301555873601e681f6fb6: Status 404 returned error can't find the container with id 16ac41cc40c6ccde8cbc41abaf646f2cdc1ed594fc2301555873601e681f6fb6
Jul 17 13:27:12.612838 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:12.612798    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6" event={"ID":"48ff745d-6aa9-441b-b52a-d50de67bda38","Type":"ContainerStarted","Data":"0fc1741cdfea2b106729cc75673282ba5dfdac0a5f88c70b20f4e978d87803a7"}
Jul 17 13:27:12.613178 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:12.612890    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6" event={"ID":"48ff745d-6aa9-441b-b52a-d50de67bda38","Type":"ContainerStarted","Data":"16ac41cc40c6ccde8cbc41abaf646f2cdc1ed594fc2301555873601e681f6fb6"}
Jul 17 13:27:12.628559 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:12.628507    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6" podStartSLOduration=1.628489122 podStartE2EDuration="1.628489122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:27:12.628467188 +0000 UTC m=+3907.632514069" watchObservedRunningTime="2026-07-17 13:27:12.628489122 +0000 UTC m=+3907.632536005"
Jul 17 13:27:13.520221 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:13.520114    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 13:27:13.535461 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:13.527681    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 13:27:13.619301 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:13.619277    2569 generic.go:356] "Generic (PLEG): container finished" podID="48ff745d-6aa9-441b-b52a-d50de67bda38" containerID="0fc1741cdfea2b106729cc75673282ba5dfdac0a5f88c70b20f4e978d87803a7" exitCode=6
Jul 17 13:27:13.619663 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:13.619364    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6" event={"ID":"48ff745d-6aa9-441b-b52a-d50de67bda38","Type":"ContainerDied","Data":"0fc1741cdfea2b106729cc75673282ba5dfdac0a5f88c70b20f4e978d87803a7"}
Jul 17 13:27:13.619663 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:13.619530    2569 scope.go:117] "RemoveContainer" containerID="0fc1741cdfea2b106729cc75673282ba5dfdac0a5f88c70b20f4e978d87803a7"
Jul 17 13:27:14.626156 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:14.626128    2569 generic.go:356] "Generic (PLEG): container finished" podID="48ff745d-6aa9-441b-b52a-d50de67bda38" containerID="df7931a9ae5cf15cad984ae4ed94293d209c26b134ebca2bf61c89dff91a5eea" exitCode=6
Jul 17 13:27:14.626577 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:14.626164    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6" event={"ID":"48ff745d-6aa9-441b-b52a-d50de67bda38","Type":"ContainerDied","Data":"df7931a9ae5cf15cad984ae4ed94293d209c26b134ebca2bf61c89dff91a5eea"}
Jul 17 13:27:14.626577 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:14.626209    2569 scope.go:117] "RemoveContainer" containerID="0fc1741cdfea2b106729cc75673282ba5dfdac0a5f88c70b20f4e978d87803a7"
Jul 17 13:27:14.626577 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:14.626436    2569 scope.go:117] "RemoveContainer" containerID="df7931a9ae5cf15cad984ae4ed94293d209c26b134ebca2bf61c89dff91a5eea"
Jul 17 13:27:14.626793 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:27:14.626651    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6_odh-ai-gateway-infra(48ff745d-6aa9-441b-b52a-d50de67bda38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6" podUID="48ff745d-6aa9-441b-b52a-d50de67bda38"
Jul 17 13:27:15.633296 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:15.633268    2569 scope.go:117] "RemoveContainer" containerID="df7931a9ae5cf15cad984ae4ed94293d209c26b134ebca2bf61c89dff91a5eea"
Jul 17 13:27:15.633685 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:27:15.633485    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6_odh-ai-gateway-infra(48ff745d-6aa9-441b-b52a-d50de67bda38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6" podUID="48ff745d-6aa9-441b-b52a-d50de67bda38"
Jul 17 13:27:17.628235 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:17.628199    2569 scope.go:117] "RemoveContainer" containerID="f02287275f3369940302cad3a18b5fde0602a7a884cd846cf65fe4900307e8c1"
Jul 17 13:27:18.649785 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:18.649737    2569 generic.go:356] "Generic (PLEG): container finished" podID="72e2017e-92fe-4a8b-bd19-13da2e0f2407" containerID="de84c537cacb72bf773662d3e42ed4969cc29a49565a6d7a6d7dc460bfe018ec" exitCode=6
Jul 17 13:27:18.650177 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:18.649796    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx" event={"ID":"72e2017e-92fe-4a8b-bd19-13da2e0f2407","Type":"ContainerDied","Data":"de84c537cacb72bf773662d3e42ed4969cc29a49565a6d7a6d7dc460bfe018ec"}
Jul 17 13:27:18.650177 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:18.649860    2569 scope.go:117] "RemoveContainer" containerID="f02287275f3369940302cad3a18b5fde0602a7a884cd846cf65fe4900307e8c1"
Jul 17 13:27:18.650177 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:18.650127    2569 scope.go:117] "RemoveContainer" containerID="de84c537cacb72bf773662d3e42ed4969cc29a49565a6d7a6d7dc460bfe018ec"
Jul 17 13:27:18.650350 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:27:18.650302    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx_odh-ai-gateway-infra(72e2017e-92fe-4a8b-bd19-13da2e0f2407)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx" podUID="72e2017e-92fe-4a8b-bd19-13da2e0f2407"
Jul 17 13:27:19.675655 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:19.675621    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx"]
Jul 17 13:27:19.808257 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:19.808235    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx"
Jul 17 13:27:19.892941 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:19.892897    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72e2017e-92fe-4a8b-bd19-13da2e0f2407-maas-api-service-ca\") pod \"72e2017e-92fe-4a8b-bd19-13da2e0f2407\" (UID: \"72e2017e-92fe-4a8b-bd19-13da2e0f2407\") "
Jul 17 13:27:19.893209 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:19.893188    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e2017e-92fe-4a8b-bd19-13da2e0f2407-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "72e2017e-92fe-4a8b-bd19-13da2e0f2407" (UID: "72e2017e-92fe-4a8b-bd19-13da2e0f2407"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:27:19.994074 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:19.994052    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72e2017e-92fe-4a8b-bd19-13da2e0f2407-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:27:20.662257 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:20.662223    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx"
Jul 17 13:27:20.662405 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:20.662281    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx" event={"ID":"72e2017e-92fe-4a8b-bd19-13da2e0f2407","Type":"ContainerDied","Data":"892f5ca0797a38742a772363468977f1455c53729bb821da92c927cf523ebe29"}
Jul 17 13:27:20.662405 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:20.662341    2569 scope.go:117] "RemoveContainer" containerID="de84c537cacb72bf773662d3e42ed4969cc29a49565a6d7a6d7dc460bfe018ec"
Jul 17 13:27:20.684507 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:20.684482    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx"]
Jul 17 13:27:20.688156 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:20.688128    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wr8xx"]
Jul 17 13:27:21.634986 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:21.634961    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72e2017e-92fe-4a8b-bd19-13da2e0f2407" path="/var/lib/kubelet/pods/72e2017e-92fe-4a8b-bd19-13da2e0f2407/volumes"
Jul 17 13:27:25.167263 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:25.167227    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:27:26.628438 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:26.628405    2569 scope.go:117] "RemoveContainer" containerID="df7931a9ae5cf15cad984ae4ed94293d209c26b134ebca2bf61c89dff91a5eea"
Jul 17 13:27:27.696645 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:27.696569    2569 generic.go:356] "Generic (PLEG): container finished" podID="48ff745d-6aa9-441b-b52a-d50de67bda38" containerID="9fcff6ad1ca4a47bc3b788adaf2a18f9c2a4656f184ade8c1cc6342aa4833f31" exitCode=6
Jul 17 13:27:27.697048 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:27.696642    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6" event={"ID":"48ff745d-6aa9-441b-b52a-d50de67bda38","Type":"ContainerDied","Data":"9fcff6ad1ca4a47bc3b788adaf2a18f9c2a4656f184ade8c1cc6342aa4833f31"}
Jul 17 13:27:27.697048 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:27.696704    2569 scope.go:117] "RemoveContainer" containerID="df7931a9ae5cf15cad984ae4ed94293d209c26b134ebca2bf61c89dff91a5eea"
Jul 17 13:27:27.697048 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:27.697034    2569 scope.go:117] "RemoveContainer" containerID="9fcff6ad1ca4a47bc3b788adaf2a18f9c2a4656f184ade8c1cc6342aa4833f31"
Jul 17 13:27:27.697224 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:27:27.697207    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6_odh-ai-gateway-infra(48ff745d-6aa9-441b-b52a-d50de67bda38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6" podUID="48ff745d-6aa9-441b-b52a-d50de67bda38"
Jul 17 13:27:28.723226 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:28.723191    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6"]
Jul 17 13:27:28.853946 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:28.853917    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6"
Jul 17 13:27:28.959061 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:28.959031    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ff745d-6aa9-441b-b52a-d50de67bda38-maas-api-service-ca\") pod \"48ff745d-6aa9-441b-b52a-d50de67bda38\" (UID: \"48ff745d-6aa9-441b-b52a-d50de67bda38\") "
Jul 17 13:27:28.959364 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:28.959339    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ff745d-6aa9-441b-b52a-d50de67bda38-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48ff745d-6aa9-441b-b52a-d50de67bda38" (UID: "48ff745d-6aa9-441b-b52a-d50de67bda38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:27:29.060183 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:29.060153    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ff745d-6aa9-441b-b52a-d50de67bda38-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:27:29.711613 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:29.711583    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6"
Jul 17 13:27:29.711779 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:29.711577    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6" event={"ID":"48ff745d-6aa9-441b-b52a-d50de67bda38","Type":"ContainerDied","Data":"16ac41cc40c6ccde8cbc41abaf646f2cdc1ed594fc2301555873601e681f6fb6"}
Jul 17 13:27:29.711779 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:29.711719    2569 scope.go:117] "RemoveContainer" containerID="9fcff6ad1ca4a47bc3b788adaf2a18f9c2a4656f184ade8c1cc6342aa4833f31"
Jul 17 13:27:29.729880 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:29.729857    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6"]
Jul 17 13:27:29.734426 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:29.734404    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-qlmj6"]
Jul 17 13:27:31.634026 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:31.633991    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48ff745d-6aa9-441b-b52a-d50de67bda38" path="/var/lib/kubelet/pods/48ff745d-6aa9-441b-b52a-d50de67bda38/volumes"
Jul 17 13:27:44.964795 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.964739    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"]
Jul 17 13:27:44.965245 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.965102    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e2017e-92fe-4a8b-bd19-13da2e0f2407" containerName="revoke-keys"
Jul 17 13:27:44.965245 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.965114    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e2017e-92fe-4a8b-bd19-13da2e0f2407" containerName="revoke-keys"
Jul 17 13:27:44.965245 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.965122    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ff745d-6aa9-441b-b52a-d50de67bda38" containerName="revoke-keys"
Jul 17 13:27:44.965245 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.965128    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ff745d-6aa9-441b-b52a-d50de67bda38" containerName="revoke-keys"
Jul 17 13:27:44.965245 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.965137    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ff745d-6aa9-441b-b52a-d50de67bda38" containerName="revoke-keys"
Jul 17 13:27:44.965245 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.965142    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ff745d-6aa9-441b-b52a-d50de67bda38" containerName="revoke-keys"
Jul 17 13:27:44.965245 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.965157    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e2017e-92fe-4a8b-bd19-13da2e0f2407" containerName="revoke-keys"
Jul 17 13:27:44.965245 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.965162    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e2017e-92fe-4a8b-bd19-13da2e0f2407" containerName="revoke-keys"
Jul 17 13:27:44.965245 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.965219    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e2017e-92fe-4a8b-bd19-13da2e0f2407" containerName="revoke-keys"
Jul 17 13:27:44.965245 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.965227    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ff745d-6aa9-441b-b52a-d50de67bda38" containerName="revoke-keys"
Jul 17 13:27:44.965245 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.965233    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ff745d-6aa9-441b-b52a-d50de67bda38" containerName="revoke-keys"
Jul 17 13:27:44.965245 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.965239    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e2017e-92fe-4a8b-bd19-13da2e0f2407" containerName="revoke-keys"
Jul 17 13:27:44.965245 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.965246    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ff745d-6aa9-441b-b52a-d50de67bda38" containerName="revoke-keys"
Jul 17 13:27:44.969685 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.969652    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:44.973669 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.973627    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5c3765\"/\"kube-root-ca.crt\""
Jul 17 13:27:44.973669 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.973643    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5c3765\"/\"openshift-service-ca.crt\""
Jul 17 13:27:44.973887 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.973689    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5c3765\"/\"auth-test-model-5c3765-kserve-self-signed-certs\""
Jul 17 13:27:44.973953 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.973941    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5c3765\"/\"default-dockercfg-dl6rb\""
Jul 17 13:27:44.975741 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:44.975718    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"]
Jul 17 13:27:45.081359 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.081334    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6499aaf4-dc4a-47f7-a100-45d895ee03be-tls-certs\") pod \"auth-test-model-5c3765-kserve-7c96f69955-8gxwt\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") " pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.081471 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.081363    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-home\") pod \"auth-test-model-5c3765-kserve-7c96f69955-8gxwt\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") " pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.081471 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.081381    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-tmp-dir\") pod \"auth-test-model-5c3765-kserve-7c96f69955-8gxwt\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") " pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.081603 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.081499    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-model-cache\") pod \"auth-test-model-5c3765-kserve-7c96f69955-8gxwt\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") " pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.081603 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.081556    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-dshm\") pod \"auth-test-model-5c3765-kserve-7c96f69955-8gxwt\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") " pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.081700 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.081599    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n96cd\" (UniqueName: \"kubernetes.io/projected/6499aaf4-dc4a-47f7-a100-45d895ee03be-kube-api-access-n96cd\") pod \"auth-test-model-5c3765-kserve-7c96f69955-8gxwt\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") " pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.182902 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.182876    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-model-cache\") pod \"auth-test-model-5c3765-kserve-7c96f69955-8gxwt\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") " pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.183019 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.182921    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-dshm\") pod \"auth-test-model-5c3765-kserve-7c96f69955-8gxwt\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") " pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.183019 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.182962    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n96cd\" (UniqueName: \"kubernetes.io/projected/6499aaf4-dc4a-47f7-a100-45d895ee03be-kube-api-access-n96cd\") pod \"auth-test-model-5c3765-kserve-7c96f69955-8gxwt\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") " pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.183019 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.183006    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6499aaf4-dc4a-47f7-a100-45d895ee03be-tls-certs\") pod \"auth-test-model-5c3765-kserve-7c96f69955-8gxwt\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") " pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.183186 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.183125    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-home\") pod \"auth-test-model-5c3765-kserve-7c96f69955-8gxwt\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") " pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.183186 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.183161    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-tmp-dir\") pod \"auth-test-model-5c3765-kserve-7c96f69955-8gxwt\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") " pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.183306 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.183214    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-model-cache\") pod \"auth-test-model-5c3765-kserve-7c96f69955-8gxwt\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") " pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.183439 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.183421    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-home\") pod \"auth-test-model-5c3765-kserve-7c96f69955-8gxwt\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") " pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.183492 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.183470    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-tmp-dir\") pod \"auth-test-model-5c3765-kserve-7c96f69955-8gxwt\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") " pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.185085 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.185065    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-dshm\") pod \"auth-test-model-5c3765-kserve-7c96f69955-8gxwt\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") " pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.185487 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.185467    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6499aaf4-dc4a-47f7-a100-45d895ee03be-tls-certs\") pod \"auth-test-model-5c3765-kserve-7c96f69955-8gxwt\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") " pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.190853 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.190836    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n96cd\" (UniqueName: \"kubernetes.io/projected/6499aaf4-dc4a-47f7-a100-45d895ee03be-kube-api-access-n96cd\") pod \"auth-test-model-5c3765-kserve-7c96f69955-8gxwt\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") " pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.280156 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.280133    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:45.409109 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.409082    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"]
Jul 17 13:27:45.410795 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:27:45.410745    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6499aaf4_dc4a_47f7_a100_45d895ee03be.slice/crio-c46ddd33a3ab8d4c14f8c749c82f9969db261997bab1b256ff8b6cf9c219add8 WatchSource:0}: Error finding container c46ddd33a3ab8d4c14f8c749c82f9969db261997bab1b256ff8b6cf9c219add8: Status 404 returned error can't find the container with id c46ddd33a3ab8d4c14f8c749c82f9969db261997bab1b256ff8b6cf9c219add8
Jul 17 13:27:45.786974 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.786929    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt" event={"ID":"6499aaf4-dc4a-47f7-a100-45d895ee03be","Type":"ContainerStarted","Data":"37127cc04e22e3dc3c99c4efc355df201f4efc15ada1984f0cb35d7bc6db2be9"}
Jul 17 13:27:45.786974 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.786981    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt" event={"ID":"6499aaf4-dc4a-47f7-a100-45d895ee03be","Type":"ContainerStarted","Data":"c46ddd33a3ab8d4c14f8c749c82f9969db261997bab1b256ff8b6cf9c219add8"}
Jul 17 13:27:45.803622 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:45.803572    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt" podStartSLOduration=1.803547367 podStartE2EDuration="1.803547367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:27:45.803294456 +0000 UTC m=+3940.807341356" watchObservedRunningTime="2026-07-17 13:27:45.803547367 +0000 UTC m=+3940.807594252"
Jul 17 13:27:48.903631 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:48.903597    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:27:55.280338 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:55.280300    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:55.280338 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:55.280354    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:55.295122 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:55.295092    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:27:55.847656 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:27:55.847619    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:28:07.877217 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:28:07.877181    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:28:27.779688 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:28:27.779648    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:28:42.661186 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:28:42.661143    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:29:01.661773 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:29:01.661718    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:29:19.815487 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:29:19.815446    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:29:34.573213 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:29:34.573160    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:29:49.074915 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:29:49.074884    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"]
Jul 17 13:29:50.378922 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:29:50.378885    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:30:04.095204 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.095150    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt" podUID="6499aaf4-dc4a-47f7-a100-45d895ee03be" containerName="main" containerID="cri-o://37127cc04e22e3dc3c99c4efc355df201f4efc15ada1984f0cb35d7bc6db2be9" gracePeriod=45
Jul 17 13:30:04.334036 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.334011    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:30:04.435046 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.434985    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-model-cache\") pod \"6499aaf4-dc4a-47f7-a100-45d895ee03be\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") "
Jul 17 13:30:04.435046 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.435039    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n96cd\" (UniqueName: \"kubernetes.io/projected/6499aaf4-dc4a-47f7-a100-45d895ee03be-kube-api-access-n96cd\") pod \"6499aaf4-dc4a-47f7-a100-45d895ee03be\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") "
Jul 17 13:30:04.435200 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.435074    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-tmp-dir\") pod \"6499aaf4-dc4a-47f7-a100-45d895ee03be\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") "
Jul 17 13:30:04.435200 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.435127    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6499aaf4-dc4a-47f7-a100-45d895ee03be-tls-certs\") pod \"6499aaf4-dc4a-47f7-a100-45d895ee03be\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") "
Jul 17 13:30:04.435200 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.435157    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-home\") pod \"6499aaf4-dc4a-47f7-a100-45d895ee03be\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") "
Jul 17 13:30:04.435200 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.435179    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-dshm\") pod \"6499aaf4-dc4a-47f7-a100-45d895ee03be\" (UID: \"6499aaf4-dc4a-47f7-a100-45d895ee03be\") "
Jul 17 13:30:04.435405 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.435264    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-model-cache" (OuterVolumeSpecName: "model-cache") pod "6499aaf4-dc4a-47f7-a100-45d895ee03be" (UID: "6499aaf4-dc4a-47f7-a100-45d895ee03be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:30:04.435405 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.435302    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6499aaf4-dc4a-47f7-a100-45d895ee03be" (UID: "6499aaf4-dc4a-47f7-a100-45d895ee03be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:30:04.435514 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.435424    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-model-cache\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:30:04.435514 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.435437    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-tmp-dir\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:30:04.435514 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.435451    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-home" (OuterVolumeSpecName: "home") pod "6499aaf4-dc4a-47f7-a100-45d895ee03be" (UID: "6499aaf4-dc4a-47f7-a100-45d895ee03be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:30:04.437447 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.437418    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-dshm" (OuterVolumeSpecName: "dshm") pod "6499aaf4-dc4a-47f7-a100-45d895ee03be" (UID: "6499aaf4-dc4a-47f7-a100-45d895ee03be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:30:04.437557 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.437459    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6499aaf4-dc4a-47f7-a100-45d895ee03be-kube-api-access-n96cd" (OuterVolumeSpecName: "kube-api-access-n96cd") pod "6499aaf4-dc4a-47f7-a100-45d895ee03be" (UID: "6499aaf4-dc4a-47f7-a100-45d895ee03be"). InnerVolumeSpecName "kube-api-access-n96cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:30:04.437557 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.437482    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6499aaf4-dc4a-47f7-a100-45d895ee03be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6499aaf4-dc4a-47f7-a100-45d895ee03be" (UID: "6499aaf4-dc4a-47f7-a100-45d895ee03be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:30:04.483433 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.483405    2569 generic.go:356] "Generic (PLEG): container finished" podID="6499aaf4-dc4a-47f7-a100-45d895ee03be" containerID="37127cc04e22e3dc3c99c4efc355df201f4efc15ada1984f0cb35d7bc6db2be9" exitCode=0
Jul 17 13:30:04.483524 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.483443    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"
Jul 17 13:30:04.483524 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.483459    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt" event={"ID":"6499aaf4-dc4a-47f7-a100-45d895ee03be","Type":"ContainerDied","Data":"37127cc04e22e3dc3c99c4efc355df201f4efc15ada1984f0cb35d7bc6db2be9"}
Jul 17 13:30:04.483524 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.483497    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt" event={"ID":"6499aaf4-dc4a-47f7-a100-45d895ee03be","Type":"ContainerDied","Data":"c46ddd33a3ab8d4c14f8c749c82f9969db261997bab1b256ff8b6cf9c219add8"}
Jul 17 13:30:04.483524 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.483518    2569 scope.go:117] "RemoveContainer" containerID="37127cc04e22e3dc3c99c4efc355df201f4efc15ada1984f0cb35d7bc6db2be9"
Jul 17 13:30:04.492987 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.492973    2569 scope.go:117] "RemoveContainer" containerID="37127cc04e22e3dc3c99c4efc355df201f4efc15ada1984f0cb35d7bc6db2be9"
Jul 17 13:30:04.493250 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:30:04.493229    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37127cc04e22e3dc3c99c4efc355df201f4efc15ada1984f0cb35d7bc6db2be9\": container with ID starting with 37127cc04e22e3dc3c99c4efc355df201f4efc15ada1984f0cb35d7bc6db2be9 not found: ID does not exist" containerID="37127cc04e22e3dc3c99c4efc355df201f4efc15ada1984f0cb35d7bc6db2be9"
Jul 17 13:30:04.493343 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.493255    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37127cc04e22e3dc3c99c4efc355df201f4efc15ada1984f0cb35d7bc6db2be9"} err="failed to get container status \"37127cc04e22e3dc3c99c4efc355df201f4efc15ada1984f0cb35d7bc6db2be9\": rpc error: code = NotFound desc = could not find container \"37127cc04e22e3dc3c99c4efc355df201f4efc15ada1984f0cb35d7bc6db2be9\": container with ID starting with 37127cc04e22e3dc3c99c4efc355df201f4efc15ada1984f0cb35d7bc6db2be9 not found: ID does not exist"
Jul 17 13:30:04.512942 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.512915    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"]
Jul 17 13:30:04.513044 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.512960    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5c3765/auth-test-model-5c3765-kserve-7c96f69955-8gxwt"]
Jul 17 13:30:04.536747 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.536727    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n96cd\" (UniqueName: \"kubernetes.io/projected/6499aaf4-dc4a-47f7-a100-45d895ee03be-kube-api-access-n96cd\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:30:04.536864 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.536762    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6499aaf4-dc4a-47f7-a100-45d895ee03be-tls-certs\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:30:04.536864 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.536777    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-home\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:30:04.536864 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:04.536785    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6499aaf4-dc4a-47f7-a100-45d895ee03be-dshm\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:30:05.633929 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:05.633901    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6499aaf4-dc4a-47f7-a100-45d895ee03be" path="/var/lib/kubelet/pods/6499aaf4-dc4a-47f7-a100-45d895ee03be/volumes"
Jul 17 13:30:08.977597 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:08.977562    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:30:20.174531 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:20.174488    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:30:36.961662 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:36.961619    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:30:41.718162 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:41.718126    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6"]
Jul 17 13:30:41.718680 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:41.718657    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e2017e-92fe-4a8b-bd19-13da2e0f2407" containerName="revoke-keys"
Jul 17 13:30:41.718815 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:41.718684    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e2017e-92fe-4a8b-bd19-13da2e0f2407" containerName="revoke-keys"
Jul 17 13:30:41.718815 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:41.718713    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6499aaf4-dc4a-47f7-a100-45d895ee03be" containerName="main"
Jul 17 13:30:41.718815 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:41.718721    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6499aaf4-dc4a-47f7-a100-45d895ee03be" containerName="main"
Jul 17 13:30:41.718815 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:41.718737    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ff745d-6aa9-441b-b52a-d50de67bda38" containerName="revoke-keys"
Jul 17 13:30:41.718815 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:41.718746    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ff745d-6aa9-441b-b52a-d50de67bda38" containerName="revoke-keys"
Jul 17 13:30:41.719091 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:41.718857    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6499aaf4-dc4a-47f7-a100-45d895ee03be" containerName="main"
Jul 17 13:30:41.719091 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:41.718874    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e2017e-92fe-4a8b-bd19-13da2e0f2407" containerName="revoke-keys"
Jul 17 13:30:41.722090 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:41.722065    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6"
Jul 17 13:30:41.724874 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:41.724847    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:30:41.729482 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:41.729460    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6"]
Jul 17 13:30:41.823339 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:41.823316    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a37c0b-b123-46de-a34e-427e6674ef82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6\" (UID: \"24a37c0b-b123-46de-a34e-427e6674ef82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6"
Jul 17 13:30:41.924251 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:41.924223    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a37c0b-b123-46de-a34e-427e6674ef82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6\" (UID: \"24a37c0b-b123-46de-a34e-427e6674ef82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6"
Jul 17 13:30:41.924879 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:41.924856    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a37c0b-b123-46de-a34e-427e6674ef82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6\" (UID: \"24a37c0b-b123-46de-a34e-427e6674ef82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6"
Jul 17 13:30:42.034385 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:42.034363    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6"
Jul 17 13:30:42.163135 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:42.163043    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6"]
Jul 17 13:30:42.165426 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:30:42.165398    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a37c0b_b123_46de_a34e_427e6674ef82.slice/crio-4cd0eb3056a073a5f576bf6d9051cab85d03a30bc6873a273295ad24a31152f2 WatchSource:0}: Error finding container 4cd0eb3056a073a5f576bf6d9051cab85d03a30bc6873a273295ad24a31152f2: Status 404 returned error can't find the container with id 4cd0eb3056a073a5f576bf6d9051cab85d03a30bc6873a273295ad24a31152f2
Jul 17 13:30:42.662634 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:42.662565    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6" event={"ID":"24a37c0b-b123-46de-a34e-427e6674ef82","Type":"ContainerStarted","Data":"d38e2c6fd35f15617092188a90ba6cfdfba2f810ea719ea674b04ebdf5a5b981"}
Jul 17 13:30:42.662634 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:42.662611    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6" event={"ID":"24a37c0b-b123-46de-a34e-427e6674ef82","Type":"ContainerStarted","Data":"4cd0eb3056a073a5f576bf6d9051cab85d03a30bc6873a273295ad24a31152f2"}
Jul 17 13:30:42.678349 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:42.678310    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6" podStartSLOduration=1.67829431 podStartE2EDuration="1.67829431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:30:42.676631931 +0000 UTC m=+4117.680678813" watchObservedRunningTime="2026-07-17 13:30:42.67829431 +0000 UTC m=+4117.682341204"
Jul 17 13:30:43.667704 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:43.667671    2569 generic.go:356] "Generic (PLEG): container finished" podID="24a37c0b-b123-46de-a34e-427e6674ef82" containerID="d38e2c6fd35f15617092188a90ba6cfdfba2f810ea719ea674b04ebdf5a5b981" exitCode=6
Jul 17 13:30:43.668103 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:43.667780    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6" event={"ID":"24a37c0b-b123-46de-a34e-427e6674ef82","Type":"ContainerDied","Data":"d38e2c6fd35f15617092188a90ba6cfdfba2f810ea719ea674b04ebdf5a5b981"}
Jul 17 13:30:43.668103 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:43.667892    2569 scope.go:117] "RemoveContainer" containerID="d38e2c6fd35f15617092188a90ba6cfdfba2f810ea719ea674b04ebdf5a5b981"
Jul 17 13:30:44.674666 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:44.674600    2569 generic.go:356] "Generic (PLEG): container finished" podID="24a37c0b-b123-46de-a34e-427e6674ef82" containerID="08d4bc7033f641e09b3fad425fc12b8c9e281e7f2be54b94a4e9d06aab660351" exitCode=6
Jul 17 13:30:44.675027 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:44.674689    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6" event={"ID":"24a37c0b-b123-46de-a34e-427e6674ef82","Type":"ContainerDied","Data":"08d4bc7033f641e09b3fad425fc12b8c9e281e7f2be54b94a4e9d06aab660351"}
Jul 17 13:30:44.675027 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:44.674779    2569 scope.go:117] "RemoveContainer" containerID="d38e2c6fd35f15617092188a90ba6cfdfba2f810ea719ea674b04ebdf5a5b981"
Jul 17 13:30:44.675027 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:44.674939    2569 scope.go:117] "RemoveContainer" containerID="08d4bc7033f641e09b3fad425fc12b8c9e281e7f2be54b94a4e9d06aab660351"
Jul 17 13:30:44.675198 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:30:44.675115    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6_odh-ai-gateway-infra(24a37c0b-b123-46de-a34e-427e6674ef82)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6" podUID="24a37c0b-b123-46de-a34e-427e6674ef82"
Jul 17 13:30:45.681321 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:45.681286    2569 scope.go:117] "RemoveContainer" containerID="08d4bc7033f641e09b3fad425fc12b8c9e281e7f2be54b94a4e9d06aab660351"
Jul 17 13:30:45.681788 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:30:45.681480    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6_odh-ai-gateway-infra(24a37c0b-b123-46de-a34e-427e6674ef82)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6" podUID="24a37c0b-b123-46de-a34e-427e6674ef82"
Jul 17 13:30:53.071701 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:53.071664    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks"]
Jul 17 13:30:53.075828 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:53.075803    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks"
Jul 17 13:30:53.082436 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:53.082407    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks"]
Jul 17 13:30:53.209990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:53.209961    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks\" (UID: \"aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks"
Jul 17 13:30:53.310809 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:53.310774    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks\" (UID: \"aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks"
Jul 17 13:30:53.311352 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:53.311333    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks\" (UID: \"aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks"
Jul 17 13:30:53.388133 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:53.388082    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks"
Jul 17 13:30:53.513291 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:53.513260    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks"]
Jul 17 13:30:53.515560 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:30:53.515536    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaae7ea3_54c4_4ce7_92ff_2a7b6575edaf.slice/crio-7f21ec5efc21a67cb0b66a2d8d5ef2ee9c7a002e7cd2042aa697e42dd2250d0f WatchSource:0}: Error finding container 7f21ec5efc21a67cb0b66a2d8d5ef2ee9c7a002e7cd2042aa697e42dd2250d0f: Status 404 returned error can't find the container with id 7f21ec5efc21a67cb0b66a2d8d5ef2ee9c7a002e7cd2042aa697e42dd2250d0f
Jul 17 13:30:53.719267 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:53.719201    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks" event={"ID":"aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf","Type":"ContainerStarted","Data":"7f21ec5efc21a67cb0b66a2d8d5ef2ee9c7a002e7cd2042aa697e42dd2250d0f"}
Jul 17 13:30:54.725350 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:54.725316    2569 generic.go:356] "Generic (PLEG): container finished" podID="aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf" containerID="f50260947667948e922f21ade75d11a67af03fb4cbe00f0eaba498c24575e21e" exitCode=6
Jul 17 13:30:54.725708 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:54.725378    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks" event={"ID":"aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf","Type":"ContainerDied","Data":"f50260947667948e922f21ade75d11a67af03fb4cbe00f0eaba498c24575e21e"}
Jul 17 13:30:54.725708 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:54.725537    2569 scope.go:117] "RemoveContainer" containerID="f50260947667948e922f21ade75d11a67af03fb4cbe00f0eaba498c24575e21e"
Jul 17 13:30:55.730793 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:55.730743    2569 generic.go:356] "Generic (PLEG): container finished" podID="aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf" containerID="c6ca5eb5e562b65dbe9f4fefb03b33894413413071fffc3e57144fce1b043c71" exitCode=6
Jul 17 13:30:55.731160 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:55.730849    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks" event={"ID":"aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf","Type":"ContainerDied","Data":"c6ca5eb5e562b65dbe9f4fefb03b33894413413071fffc3e57144fce1b043c71"}
Jul 17 13:30:55.731160 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:55.730895    2569 scope.go:117] "RemoveContainer" containerID="f50260947667948e922f21ade75d11a67af03fb4cbe00f0eaba498c24575e21e"
Jul 17 13:30:55.731160 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:55.731007    2569 scope.go:117] "RemoveContainer" containerID="c6ca5eb5e562b65dbe9f4fefb03b33894413413071fffc3e57144fce1b043c71"
Jul 17 13:30:55.731334 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:30:55.731201    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks_odh-ai-gateway-infra(aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks" podUID="aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf"
Jul 17 13:30:56.736421 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:56.736393    2569 scope.go:117] "RemoveContainer" containerID="c6ca5eb5e562b65dbe9f4fefb03b33894413413071fffc3e57144fce1b043c71"
Jul 17 13:30:56.736881 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:30:56.736595    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks_odh-ai-gateway-infra(aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks" podUID="aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf"
Jul 17 13:30:58.568114 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:30:58.568022    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:31:01.629303 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:01.629261    2569 scope.go:117] "RemoveContainer" containerID="08d4bc7033f641e09b3fad425fc12b8c9e281e7f2be54b94a4e9d06aab660351"
Jul 17 13:31:02.766995 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.766967    2569 generic.go:356] "Generic (PLEG): container finished" podID="24a37c0b-b123-46de-a34e-427e6674ef82" containerID="c750193be0a9cce87c2bb316c994405d43384819855db0b00b19ff8f7bbc1ed2" exitCode=6
Jul 17 13:31:02.767421 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.767048    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6" event={"ID":"24a37c0b-b123-46de-a34e-427e6674ef82","Type":"ContainerDied","Data":"c750193be0a9cce87c2bb316c994405d43384819855db0b00b19ff8f7bbc1ed2"}
Jul 17 13:31:02.767421 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.767128    2569 scope.go:117] "RemoveContainer" containerID="08d4bc7033f641e09b3fad425fc12b8c9e281e7f2be54b94a4e9d06aab660351"
Jul 17 13:31:02.767566 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.767538    2569 scope.go:117] "RemoveContainer" containerID="c750193be0a9cce87c2bb316c994405d43384819855db0b00b19ff8f7bbc1ed2"
Jul 17 13:31:02.767934 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:31:02.767741    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6_odh-ai-gateway-infra(24a37c0b-b123-46de-a34e-427e6674ef82)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6" podUID="24a37c0b-b123-46de-a34e-427e6674ef82"
Jul 17 13:31:02.772376 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.772355    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"]
Jul 17 13:31:02.775699 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.775677    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:02.778148 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.778130    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e92eab\"/\"rate-test-model-e92eab-kserve-self-signed-certs\""
Jul 17 13:31:02.779299 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.779114    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e92eab\"/\"default-dockercfg-qwfzb\""
Jul 17 13:31:02.779299 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.779158    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e92eab\"/\"kube-root-ca.crt\""
Jul 17 13:31:02.779299 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.779162    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e92eab\"/\"openshift-service-ca.crt\""
Jul 17 13:31:02.785614 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.785562    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"]
Jul 17 13:31:02.887243 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.887215    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e2dbaee-296d-4d28-8207-eb935cc0c89b-tls-certs\") pod \"rate-test-model-e92eab-kserve-578795b5b7-fbnrn\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") " pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:02.887362 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.887269    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfdl7\" (UniqueName: \"kubernetes.io/projected/4e2dbaee-296d-4d28-8207-eb935cc0c89b-kube-api-access-mfdl7\") pod \"rate-test-model-e92eab-kserve-578795b5b7-fbnrn\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") " pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:02.887362 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.887315    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-tmp-dir\") pod \"rate-test-model-e92eab-kserve-578795b5b7-fbnrn\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") " pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:02.887461 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.887376    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-dshm\") pod \"rate-test-model-e92eab-kserve-578795b5b7-fbnrn\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") " pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:02.887461 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.887411    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-home\") pod \"rate-test-model-e92eab-kserve-578795b5b7-fbnrn\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") " pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:02.887461 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.887456    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-model-cache\") pod \"rate-test-model-e92eab-kserve-578795b5b7-fbnrn\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") " pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:02.988599 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.988578    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-tmp-dir\") pod \"rate-test-model-e92eab-kserve-578795b5b7-fbnrn\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") " pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:02.988706 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.988614    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-dshm\") pod \"rate-test-model-e92eab-kserve-578795b5b7-fbnrn\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") " pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:02.988706 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.988643    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-home\") pod \"rate-test-model-e92eab-kserve-578795b5b7-fbnrn\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") " pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:02.988706 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.988689    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-model-cache\") pod \"rate-test-model-e92eab-kserve-578795b5b7-fbnrn\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") " pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:02.988903 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.988739    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e2dbaee-296d-4d28-8207-eb935cc0c89b-tls-certs\") pod \"rate-test-model-e92eab-kserve-578795b5b7-fbnrn\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") " pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:02.988903 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.988826    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfdl7\" (UniqueName: \"kubernetes.io/projected/4e2dbaee-296d-4d28-8207-eb935cc0c89b-kube-api-access-mfdl7\") pod \"rate-test-model-e92eab-kserve-578795b5b7-fbnrn\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") " pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:02.989054 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.989032    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-tmp-dir\") pod \"rate-test-model-e92eab-kserve-578795b5b7-fbnrn\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") " pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:02.989102 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.989071    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-home\") pod \"rate-test-model-e92eab-kserve-578795b5b7-fbnrn\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") " pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:02.989150 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.989105    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-model-cache\") pod \"rate-test-model-e92eab-kserve-578795b5b7-fbnrn\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") " pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:02.990846 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.990826    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-dshm\") pod \"rate-test-model-e92eab-kserve-578795b5b7-fbnrn\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") " pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:02.991242 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.991222    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e2dbaee-296d-4d28-8207-eb935cc0c89b-tls-certs\") pod \"rate-test-model-e92eab-kserve-578795b5b7-fbnrn\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") " pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:02.997390 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:02.997366    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfdl7\" (UniqueName: \"kubernetes.io/projected/4e2dbaee-296d-4d28-8207-eb935cc0c89b-kube-api-access-mfdl7\") pod \"rate-test-model-e92eab-kserve-578795b5b7-fbnrn\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") " pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:03.089674 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:03.089625    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:03.221739 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:03.221716    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"]
Jul 17 13:31:03.223489 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:31:03.223462    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e2dbaee_296d_4d28_8207_eb935cc0c89b.slice/crio-7980ced8e91715a11694b14a780c779ee92cb923d91a3e6b4450855c5d0d5fb1 WatchSource:0}: Error finding container 7980ced8e91715a11694b14a780c779ee92cb923d91a3e6b4450855c5d0d5fb1: Status 404 returned error can't find the container with id 7980ced8e91715a11694b14a780c779ee92cb923d91a3e6b4450855c5d0d5fb1
Jul 17 13:31:03.772519 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:03.772481    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn" event={"ID":"4e2dbaee-296d-4d28-8207-eb935cc0c89b","Type":"ContainerStarted","Data":"6747de8955f70739b9b7a8237a33207741af68aa146318715ec2cade117fa88c"}
Jul 17 13:31:03.772519 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:03.772530    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn" event={"ID":"4e2dbaee-296d-4d28-8207-eb935cc0c89b","Type":"ContainerStarted","Data":"7980ced8e91715a11694b14a780c779ee92cb923d91a3e6b4450855c5d0d5fb1"}
Jul 17 13:31:03.790290 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:03.790242    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn" podStartSLOduration=1.790227145 podStartE2EDuration="1.790227145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:31:03.788673613 +0000 UTC m=+4138.792720495" watchObservedRunningTime="2026-07-17 13:31:03.790227145 +0000 UTC m=+4138.794274027"
Jul 17 13:31:03.798876 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:03.798853    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6"]
Jul 17 13:31:03.917346 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:03.917315    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6"
Jul 17 13:31:03.996539 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:03.996511    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a37c0b-b123-46de-a34e-427e6674ef82-maas-api-service-ca\") pod \"24a37c0b-b123-46de-a34e-427e6674ef82\" (UID: \"24a37c0b-b123-46de-a34e-427e6674ef82\") "
Jul 17 13:31:03.997072 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:03.996845    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a37c0b-b123-46de-a34e-427e6674ef82-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "24a37c0b-b123-46de-a34e-427e6674ef82" (UID: "24a37c0b-b123-46de-a34e-427e6674ef82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:31:04.097449 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:04.097393    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a37c0b-b123-46de-a34e-427e6674ef82-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:31:04.779783 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:04.779739    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6"
Jul 17 13:31:04.780208 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:04.779787    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6" event={"ID":"24a37c0b-b123-46de-a34e-427e6674ef82","Type":"ContainerDied","Data":"4cd0eb3056a073a5f576bf6d9051cab85d03a30bc6873a273295ad24a31152f2"}
Jul 17 13:31:04.780208 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:04.779887    2569 scope.go:117] "RemoveContainer" containerID="c750193be0a9cce87c2bb316c994405d43384819855db0b00b19ff8f7bbc1ed2"
Jul 17 13:31:04.800908 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:04.800886    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6"]
Jul 17 13:31:04.804601 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:04.804578    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-tkvf6"]
Jul 17 13:31:05.634315 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:05.634285    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24a37c0b-b123-46de-a34e-427e6674ef82" path="/var/lib/kubelet/pods/24a37c0b-b123-46de-a34e-427e6674ef82/volumes"
Jul 17 13:31:09.628378 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:09.628347    2569 scope.go:117] "RemoveContainer" containerID="c6ca5eb5e562b65dbe9f4fefb03b33894413413071fffc3e57144fce1b043c71"
Jul 17 13:31:10.811764 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:10.811730    2569 generic.go:356] "Generic (PLEG): container finished" podID="aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf" containerID="e0b31ed8156e9745f500e550578c2cf0be9bd409be70d776f053f510ddbd45a6" exitCode=6
Jul 17 13:31:10.812122 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:10.811800    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks" event={"ID":"aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf","Type":"ContainerDied","Data":"e0b31ed8156e9745f500e550578c2cf0be9bd409be70d776f053f510ddbd45a6"}
Jul 17 13:31:10.812122 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:10.811861    2569 scope.go:117] "RemoveContainer" containerID="c6ca5eb5e562b65dbe9f4fefb03b33894413413071fffc3e57144fce1b043c71"
Jul 17 13:31:10.812122 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:10.812098    2569 scope.go:117] "RemoveContainer" containerID="e0b31ed8156e9745f500e550578c2cf0be9bd409be70d776f053f510ddbd45a6"
Jul 17 13:31:10.812292 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:31:10.812275    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks_odh-ai-gateway-infra(aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks" podUID="aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf"
Jul 17 13:31:11.838699 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:11.838663    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks"]
Jul 17 13:31:11.973243 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:11.973214    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks"
Jul 17 13:31:12.058833 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:12.058805    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf-maas-api-service-ca\") pod \"aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf\" (UID: \"aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf\") "
Jul 17 13:31:12.059119 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:12.059099    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf" (UID: "aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:31:12.159476 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:12.159417    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:31:12.823772 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:12.823726    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks"
Jul 17 13:31:12.823772 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:12.823741    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks" event={"ID":"aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf","Type":"ContainerDied","Data":"7f21ec5efc21a67cb0b66a2d8d5ef2ee9c7a002e7cd2042aa697e42dd2250d0f"}
Jul 17 13:31:12.823979 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:12.823828    2569 scope.go:117] "RemoveContainer" containerID="e0b31ed8156e9745f500e550578c2cf0be9bd409be70d776f053f510ddbd45a6"
Jul 17 13:31:12.845245 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:12.845220    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks"]
Jul 17 13:31:12.854210 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:12.854184    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-bsvks"]
Jul 17 13:31:13.089999 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:13.089944    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:13.089999 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:13.089974    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:13.102557 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:13.102541    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:13.633744 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:13.633718    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf" path="/var/lib/kubelet/pods/aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf/volumes"
Jul 17 13:31:13.841706 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:13.841679    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:31:15.468432 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:15.468399    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:31:35.962159 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:35.962122    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:31:50.965399 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:31:50.965363    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:32:07.479695 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:07.479652    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:32:13.548585 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:13.548479    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 13:32:13.557966 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:13.557948    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-27xxg_fa1a19dd-c085-43ae-8dc2-d5775ccd5f59/ovn-acl-logging/0.log"
Jul 17 13:32:13.579985 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:13.579961    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"]
Jul 17 13:32:21.770130 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:21.770092    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:32:28.600045 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:28.599955    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn" podUID="4e2dbaee-296d-4d28-8207-eb935cc0c89b" containerName="main" containerID="cri-o://6747de8955f70739b9b7a8237a33207741af68aa146318715ec2cade117fa88c" gracePeriod=45
Jul 17 13:32:28.846271 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:28.846245    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:32:28.920483 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:28.920421    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e2dbaee-296d-4d28-8207-eb935cc0c89b-tls-certs\") pod \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") "
Jul 17 13:32:28.920483 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:28.920476    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-model-cache\") pod \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") "
Jul 17 13:32:28.920641 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:28.920499    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-tmp-dir\") pod \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") "
Jul 17 13:32:28.920641 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:28.920524    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfdl7\" (UniqueName: \"kubernetes.io/projected/4e2dbaee-296d-4d28-8207-eb935cc0c89b-kube-api-access-mfdl7\") pod \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") "
Jul 17 13:32:28.920641 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:28.920569    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-home\") pod \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") "
Jul 17 13:32:28.920641 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:28.920592    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-dshm\") pod \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\" (UID: \"4e2dbaee-296d-4d28-8207-eb935cc0c89b\") "
Jul 17 13:32:28.920882 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:28.920737    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e2dbaee-296d-4d28-8207-eb935cc0c89b" (UID: "4e2dbaee-296d-4d28-8207-eb935cc0c89b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:32:28.920882 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:28.920819    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-home" (OuterVolumeSpecName: "home") pod "4e2dbaee-296d-4d28-8207-eb935cc0c89b" (UID: "4e2dbaee-296d-4d28-8207-eb935cc0c89b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:32:28.920882 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:28.920837    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e2dbaee-296d-4d28-8207-eb935cc0c89b" (UID: "4e2dbaee-296d-4d28-8207-eb935cc0c89b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:32:28.921023 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:28.920936    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-model-cache\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:32:28.921023 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:28.920956    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-tmp-dir\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:32:28.921023 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:28.920965    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-home\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:32:28.922670 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:28.922642    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-dshm" (OuterVolumeSpecName: "dshm") pod "4e2dbaee-296d-4d28-8207-eb935cc0c89b" (UID: "4e2dbaee-296d-4d28-8207-eb935cc0c89b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 13:32:28.922785 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:28.922740    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2dbaee-296d-4d28-8207-eb935cc0c89b-kube-api-access-mfdl7" (OuterVolumeSpecName: "kube-api-access-mfdl7") pod "4e2dbaee-296d-4d28-8207-eb935cc0c89b" (UID: "4e2dbaee-296d-4d28-8207-eb935cc0c89b"). InnerVolumeSpecName "kube-api-access-mfdl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:32:28.922839 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:28.922778    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2dbaee-296d-4d28-8207-eb935cc0c89b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e2dbaee-296d-4d28-8207-eb935cc0c89b" (UID: "4e2dbaee-296d-4d28-8207-eb935cc0c89b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:32:29.021536 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:29.021514    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfdl7\" (UniqueName: \"kubernetes.io/projected/4e2dbaee-296d-4d28-8207-eb935cc0c89b-kube-api-access-mfdl7\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:32:29.021536 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:29.021537    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e2dbaee-296d-4d28-8207-eb935cc0c89b-dshm\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:32:29.021670 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:29.021546    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e2dbaee-296d-4d28-8207-eb935cc0c89b-tls-certs\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:32:29.199843 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:29.199783    2569 generic.go:356] "Generic (PLEG): container finished" podID="4e2dbaee-296d-4d28-8207-eb935cc0c89b" containerID="6747de8955f70739b9b7a8237a33207741af68aa146318715ec2cade117fa88c" exitCode=0
Jul 17 13:32:29.199843 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:29.199826    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"
Jul 17 13:32:29.199986 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:29.199868    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn" event={"ID":"4e2dbaee-296d-4d28-8207-eb935cc0c89b","Type":"ContainerDied","Data":"6747de8955f70739b9b7a8237a33207741af68aa146318715ec2cade117fa88c"}
Jul 17 13:32:29.199986 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:29.199912    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn" event={"ID":"4e2dbaee-296d-4d28-8207-eb935cc0c89b","Type":"ContainerDied","Data":"7980ced8e91715a11694b14a780c779ee92cb923d91a3e6b4450855c5d0d5fb1"}
Jul 17 13:32:29.199986 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:29.199930    2569 scope.go:117] "RemoveContainer" containerID="6747de8955f70739b9b7a8237a33207741af68aa146318715ec2cade117fa88c"
Jul 17 13:32:29.209094 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:29.209080    2569 scope.go:117] "RemoveContainer" containerID="6747de8955f70739b9b7a8237a33207741af68aa146318715ec2cade117fa88c"
Jul 17 13:32:29.209375 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:32:29.209358    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6747de8955f70739b9b7a8237a33207741af68aa146318715ec2cade117fa88c\": container with ID starting with 6747de8955f70739b9b7a8237a33207741af68aa146318715ec2cade117fa88c not found: ID does not exist" containerID="6747de8955f70739b9b7a8237a33207741af68aa146318715ec2cade117fa88c"
Jul 17 13:32:29.209414 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:29.209385    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6747de8955f70739b9b7a8237a33207741af68aa146318715ec2cade117fa88c"} err="failed to get container status \"6747de8955f70739b9b7a8237a33207741af68aa146318715ec2cade117fa88c\": rpc error: code = NotFound desc = could not find container \"6747de8955f70739b9b7a8237a33207741af68aa146318715ec2cade117fa88c\": container with ID starting with 6747de8955f70739b9b7a8237a33207741af68aa146318715ec2cade117fa88c not found: ID does not exist"
Jul 17 13:32:29.221375 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:29.221350    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"]
Jul 17 13:32:29.226839 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:29.226818    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e92eab/rate-test-model-e92eab-kserve-578795b5b7-fbnrn"]
Jul 17 13:32:29.633722 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:29.633700    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e2dbaee-296d-4d28-8207-eb935cc0c89b" path="/var/lib/kubelet/pods/4e2dbaee-296d-4d28-8207-eb935cc0c89b/volumes"
Jul 17 13:32:31.809917 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.809879    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln"]
Jul 17 13:32:31.810325 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.810250    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2dbaee-296d-4d28-8207-eb935cc0c89b" containerName="main"
Jul 17 13:32:31.810325 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.810263    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2dbaee-296d-4d28-8207-eb935cc0c89b" containerName="main"
Jul 17 13:32:31.810325 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.810284    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24a37c0b-b123-46de-a34e-427e6674ef82" containerName="revoke-keys"
Jul 17 13:32:31.810325 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.810291    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="24a37c0b-b123-46de-a34e-427e6674ef82" containerName="revoke-keys"
Jul 17 13:32:31.810325 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.810302    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf" containerName="revoke-keys"
Jul 17 13:32:31.810325 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.810308    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf" containerName="revoke-keys"
Jul 17 13:32:31.810325 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.810325    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf" containerName="revoke-keys"
Jul 17 13:32:31.810325 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.810330    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf" containerName="revoke-keys"
Jul 17 13:32:31.810617 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.810343    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24a37c0b-b123-46de-a34e-427e6674ef82" containerName="revoke-keys"
Jul 17 13:32:31.810617 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.810351    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="24a37c0b-b123-46de-a34e-427e6674ef82" containerName="revoke-keys"
Jul 17 13:32:31.810617 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.810466    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="24a37c0b-b123-46de-a34e-427e6674ef82" containerName="revoke-keys"
Jul 17 13:32:31.810617 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.810481    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf" containerName="revoke-keys"
Jul 17 13:32:31.810617 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.810493    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf" containerName="revoke-keys"
Jul 17 13:32:31.810617 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.810527    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf" containerName="revoke-keys"
Jul 17 13:32:31.810617 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.810537    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="24a37c0b-b123-46de-a34e-427e6674ef82" containerName="revoke-keys"
Jul 17 13:32:31.810617 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.810548    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2dbaee-296d-4d28-8207-eb935cc0c89b" containerName="main"
Jul 17 13:32:31.815180 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.815157    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln"
Jul 17 13:32:31.817897 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.817874    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:32:31.821942 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.821913    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln"]
Jul 17 13:32:31.940709 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:31.940685    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b304a07-e0a5-4174-a2bd-771c1f7238de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln\" (UID: \"1b304a07-e0a5-4174-a2bd-771c1f7238de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln"
Jul 17 13:32:32.041341 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:32.041314    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b304a07-e0a5-4174-a2bd-771c1f7238de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln\" (UID: \"1b304a07-e0a5-4174-a2bd-771c1f7238de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln"
Jul 17 13:32:32.041982 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:32.041955    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b304a07-e0a5-4174-a2bd-771c1f7238de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln\" (UID: \"1b304a07-e0a5-4174-a2bd-771c1f7238de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln"
Jul 17 13:32:32.127226 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:32.127167    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln"
Jul 17 13:32:32.256310 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:32.256288    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln"]
Jul 17 13:32:32.258591 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:32:32.258563    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b304a07_e0a5_4174_a2bd_771c1f7238de.slice/crio-ff1471cf138f4476f1e0cc4b0b26364a140a4f30bfac80733be9dc2f05217b53 WatchSource:0}: Error finding container ff1471cf138f4476f1e0cc4b0b26364a140a4f30bfac80733be9dc2f05217b53: Status 404 returned error can't find the container with id ff1471cf138f4476f1e0cc4b0b26364a140a4f30bfac80733be9dc2f05217b53
Jul 17 13:32:33.146949 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.146886    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf"]
Jul 17 13:32:33.147276 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.147260    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24a37c0b-b123-46de-a34e-427e6674ef82" containerName="revoke-keys"
Jul 17 13:32:33.147276 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.147276    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="24a37c0b-b123-46de-a34e-427e6674ef82" containerName="revoke-keys"
Jul 17 13:32:33.147349 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.147289    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf" containerName="revoke-keys"
Jul 17 13:32:33.147349 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.147295    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaae7ea3-54c4-4ce7-92ff-2a7b6575edaf" containerName="revoke-keys"
Jul 17 13:32:33.147414 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.147358    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="24a37c0b-b123-46de-a34e-427e6674ef82" containerName="revoke-keys"
Jul 17 13:32:33.150369 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.150351    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf"
Jul 17 13:32:33.157534 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.157509    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf"]
Jul 17 13:32:33.221934 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.221901    2569 generic.go:356] "Generic (PLEG): container finished" podID="1b304a07-e0a5-4174-a2bd-771c1f7238de" containerID="2fe62b04bbe1b4077f0178e5cbe7acad81980e872d5518dc4bf2de4ea3e6b16b" exitCode=6
Jul 17 13:32:33.222054 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.221946    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln" event={"ID":"1b304a07-e0a5-4174-a2bd-771c1f7238de","Type":"ContainerDied","Data":"2fe62b04bbe1b4077f0178e5cbe7acad81980e872d5518dc4bf2de4ea3e6b16b"}
Jul 17 13:32:33.222054 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.221998    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln" event={"ID":"1b304a07-e0a5-4174-a2bd-771c1f7238de","Type":"ContainerStarted","Data":"ff1471cf138f4476f1e0cc4b0b26364a140a4f30bfac80733be9dc2f05217b53"}
Jul 17 13:32:33.222167 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.222149    2569 scope.go:117] "RemoveContainer" containerID="2fe62b04bbe1b4077f0178e5cbe7acad81980e872d5518dc4bf2de4ea3e6b16b"
Jul 17 13:32:33.248385 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.248345    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf\" (UID: \"7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf"
Jul 17 13:32:33.349480 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.349457    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf\" (UID: \"7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf"
Jul 17 13:32:33.350037 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.350019    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf\" (UID: \"7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf"
Jul 17 13:32:33.462521 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.462493    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf"
Jul 17 13:32:33.588934 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.588901    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf"]
Jul 17 13:32:33.591988 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:32:33.591959    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2c2f0a_37d4_4a99_a28b_58d0d2bfd3b5.slice/crio-bdeb7e6bdcb05c297e1c73cb01684c54edd908c8d61cca152e9ad40eead09d4b WatchSource:0}: Error finding container bdeb7e6bdcb05c297e1c73cb01684c54edd908c8d61cca152e9ad40eead09d4b: Status 404 returned error can't find the container with id bdeb7e6bdcb05c297e1c73cb01684c54edd908c8d61cca152e9ad40eead09d4b
Jul 17 13:32:33.904990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.904964    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e92eab-69sdj"]
Jul 17 13:32:33.908205 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.908189    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e92eab-69sdj"
Jul 17 13:32:33.916823 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:33.916794    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e92eab-69sdj"]
Jul 17 13:32:34.055728 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:34.055703    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99e479e-60c6-4bb2-b497-f744efd9e071-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-e92eab-69sdj\" (UID: \"a99e479e-60c6-4bb2-b497-f744efd9e071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e92eab-69sdj"
Jul 17 13:32:34.157019 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:34.156967    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99e479e-60c6-4bb2-b497-f744efd9e071-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-e92eab-69sdj\" (UID: \"a99e479e-60c6-4bb2-b497-f744efd9e071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e92eab-69sdj"
Jul 17 13:32:34.157483 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:34.157465    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99e479e-60c6-4bb2-b497-f744efd9e071-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-e92eab-69sdj\" (UID: \"a99e479e-60c6-4bb2-b497-f744efd9e071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e92eab-69sdj"
Jul 17 13:32:34.218924 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:34.218901    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e92eab-69sdj"
Jul 17 13:32:34.231808 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:34.231617    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf" event={"ID":"7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5","Type":"ContainerStarted","Data":"0cd57a26265d8dd58a20163b9b58039a9725d45ade19aef5c901555c59f7ca9a"}
Jul 17 13:32:34.231808 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:34.231675    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf" event={"ID":"7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5","Type":"ContainerStarted","Data":"bdeb7e6bdcb05c297e1c73cb01684c54edd908c8d61cca152e9ad40eead09d4b"}
Jul 17 13:32:34.233917 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:34.233894    2569 generic.go:356] "Generic (PLEG): container finished" podID="1b304a07-e0a5-4174-a2bd-771c1f7238de" containerID="692e5b717e42b6c4ce3d94e708bc6a2962fa7276322913c97f9c9d96bfaecdef" exitCode=6
Jul 17 13:32:34.234028 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:34.233960    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln" event={"ID":"1b304a07-e0a5-4174-a2bd-771c1f7238de","Type":"ContainerDied","Data":"692e5b717e42b6c4ce3d94e708bc6a2962fa7276322913c97f9c9d96bfaecdef"}
Jul 17 13:32:34.234028 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:34.234004    2569 scope.go:117] "RemoveContainer" containerID="2fe62b04bbe1b4077f0178e5cbe7acad81980e872d5518dc4bf2de4ea3e6b16b"
Jul 17 13:32:34.234187 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:34.234166    2569 scope.go:117] "RemoveContainer" containerID="692e5b717e42b6c4ce3d94e708bc6a2962fa7276322913c97f9c9d96bfaecdef"
Jul 17 13:32:34.234387 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:32:34.234362    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln_odh-ai-gateway-infra(1b304a07-e0a5-4174-a2bd-771c1f7238de)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln" podUID="1b304a07-e0a5-4174-a2bd-771c1f7238de"
Jul 17 13:32:34.249608 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:34.249567    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf" podStartSLOduration=1.2495542259999999 podStartE2EDuration="1.249554226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:32:34.247589682 +0000 UTC m=+4229.251636565" watchObservedRunningTime="2026-07-17 13:32:34.249554226 +0000 UTC m=+4229.253601108"
Jul 17 13:32:34.350430 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:34.350404    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e92eab-69sdj"]
Jul 17 13:32:34.352071 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:32:34.352043    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99e479e_60c6_4bb2_b497_f744efd9e071.slice/crio-36346f210fb79bac1f5b022f779dfdca069905f6869b23070ba55cafa5133f91 WatchSource:0}: Error finding container 36346f210fb79bac1f5b022f779dfdca069905f6869b23070ba55cafa5133f91: Status 404 returned error can't find the container with id 36346f210fb79bac1f5b022f779dfdca069905f6869b23070ba55cafa5133f91
Jul 17 13:32:35.240568 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:35.240545    2569 scope.go:117] "RemoveContainer" containerID="692e5b717e42b6c4ce3d94e708bc6a2962fa7276322913c97f9c9d96bfaecdef"
Jul 17 13:32:35.240975 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:32:35.240720    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln_odh-ai-gateway-infra(1b304a07-e0a5-4174-a2bd-771c1f7238de)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln" podUID="1b304a07-e0a5-4174-a2bd-771c1f7238de"
Jul 17 13:32:35.242267 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:35.242238    2569 generic.go:356] "Generic (PLEG): container finished" podID="a99e479e-60c6-4bb2-b497-f744efd9e071" containerID="fd60932e24cd7d9789683cd3c44b473aeb56bbe12a3819b272d1eb7ab2c7e98f" exitCode=0
Jul 17 13:32:35.242379 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:35.242285    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e92eab-69sdj" event={"ID":"a99e479e-60c6-4bb2-b497-f744efd9e071","Type":"ContainerDied","Data":"fd60932e24cd7d9789683cd3c44b473aeb56bbe12a3819b272d1eb7ab2c7e98f"}
Jul 17 13:32:35.242379 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:35.242333    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e92eab-69sdj" event={"ID":"a99e479e-60c6-4bb2-b497-f744efd9e071","Type":"ContainerStarted","Data":"36346f210fb79bac1f5b022f779dfdca069905f6869b23070ba55cafa5133f91"}
Jul 17 13:32:35.243818 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:35.243795    2569 generic.go:356] "Generic (PLEG): container finished" podID="7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5" containerID="0cd57a26265d8dd58a20163b9b58039a9725d45ade19aef5c901555c59f7ca9a" exitCode=6
Jul 17 13:32:35.243927 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:35.243863    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf" event={"ID":"7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5","Type":"ContainerDied","Data":"0cd57a26265d8dd58a20163b9b58039a9725d45ade19aef5c901555c59f7ca9a"}
Jul 17 13:32:35.244025 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:35.244010    2569 scope.go:117] "RemoveContainer" containerID="0cd57a26265d8dd58a20163b9b58039a9725d45ade19aef5c901555c59f7ca9a"
Jul 17 13:32:36.251389 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:36.251353    2569 generic.go:356] "Generic (PLEG): container finished" podID="7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5" containerID="2b3841b4bbcb9d5d3b0c90bff9477a1a71ca97dff46b6dc0951a9ee8e6baefdb" exitCode=6
Jul 17 13:32:36.251865 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:36.251441    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf" event={"ID":"7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5","Type":"ContainerDied","Data":"2b3841b4bbcb9d5d3b0c90bff9477a1a71ca97dff46b6dc0951a9ee8e6baefdb"}
Jul 17 13:32:36.251865 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:36.251534    2569 scope.go:117] "RemoveContainer" containerID="0cd57a26265d8dd58a20163b9b58039a9725d45ade19aef5c901555c59f7ca9a"
Jul 17 13:32:36.251865 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:36.251821    2569 scope.go:117] "RemoveContainer" containerID="2b3841b4bbcb9d5d3b0c90bff9477a1a71ca97dff46b6dc0951a9ee8e6baefdb"
Jul 17 13:32:36.252411 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:32:36.252048    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf_odh-ai-gateway-infra(7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf" podUID="7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5"
Jul 17 13:32:36.387929 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:36.387908    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e92eab-69sdj"
Jul 17 13:32:36.475209 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:36.475184    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99e479e-60c6-4bb2-b497-f744efd9e071-maas-api-service-ca\") pod \"a99e479e-60c6-4bb2-b497-f744efd9e071\" (UID: \"a99e479e-60c6-4bb2-b497-f744efd9e071\") "
Jul 17 13:32:36.475547 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:36.475526    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99e479e-60c6-4bb2-b497-f744efd9e071-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a99e479e-60c6-4bb2-b497-f744efd9e071" (UID: "a99e479e-60c6-4bb2-b497-f744efd9e071"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:32:36.575868 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:36.575805    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a99e479e-60c6-4bb2-b497-f744efd9e071-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:32:37.258918 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:37.258884    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e92eab-69sdj"
Jul 17 13:32:37.259360 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:37.258889    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e92eab-69sdj" event={"ID":"a99e479e-60c6-4bb2-b497-f744efd9e071","Type":"ContainerDied","Data":"36346f210fb79bac1f5b022f779dfdca069905f6869b23070ba55cafa5133f91"}
Jul 17 13:32:37.259360 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:37.259015    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36346f210fb79bac1f5b022f779dfdca069905f6869b23070ba55cafa5133f91"
Jul 17 13:32:37.260981 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:37.260956    2569 scope.go:117] "RemoveContainer" containerID="2b3841b4bbcb9d5d3b0c90bff9477a1a71ca97dff46b6dc0951a9ee8e6baefdb"
Jul 17 13:32:37.261207 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:32:37.261183    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf_odh-ai-gateway-infra(7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf" podUID="7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5"
Jul 17 13:32:39.284144 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:39.284102    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z"]
Jul 17 13:32:39.284557 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:39.284510    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a99e479e-60c6-4bb2-b497-f744efd9e071" containerName="revoke-keys"
Jul 17 13:32:39.284557 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:39.284525    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a99e479e-60c6-4bb2-b497-f744efd9e071" containerName="revoke-keys"
Jul 17 13:32:39.284635 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:39.284612    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a99e479e-60c6-4bb2-b497-f744efd9e071" containerName="revoke-keys"
Jul 17 13:32:39.287859 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:39.287837    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z"
Jul 17 13:32:39.294511 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:39.294476    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z"]
Jul 17 13:32:39.395669 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:39.395643    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z\" (UID: \"ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z"
Jul 17 13:32:39.496384 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:39.496355    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z\" (UID: \"ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z"
Jul 17 13:32:39.496985 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:39.496965    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z\" (UID: \"ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z"
Jul 17 13:32:39.600366 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:39.600315    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z"
Jul 17 13:32:39.727512 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:39.727489    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z"]
Jul 17 13:32:39.729422 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:32:39.729394    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded0b8dc_92ef_40ce_8925_cf6ac2c1fdb9.slice/crio-a2235bf80134f6f92202766272608e24f2a9c4aa87e6decd0cef1df06f7e49aa WatchSource:0}: Error finding container a2235bf80134f6f92202766272608e24f2a9c4aa87e6decd0cef1df06f7e49aa: Status 404 returned error can't find the container with id a2235bf80134f6f92202766272608e24f2a9c4aa87e6decd0cef1df06f7e49aa
Jul 17 13:32:40.276660 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:40.276632    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z" event={"ID":"ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9","Type":"ContainerStarted","Data":"3174b55cda303dd6e1f76e4105a761534a463d5e64943b4588f90d037e415d92"}
Jul 17 13:32:40.276798 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:40.276689    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z" event={"ID":"ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9","Type":"ContainerStarted","Data":"a2235bf80134f6f92202766272608e24f2a9c4aa87e6decd0cef1df06f7e49aa"}
Jul 17 13:32:40.292541 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:40.292501    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z" podStartSLOduration=1.2924876410000001 podStartE2EDuration="1.292487641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:32:40.291589083 +0000 UTC m=+4235.295635965" watchObservedRunningTime="2026-07-17 13:32:40.292487641 +0000 UTC m=+4235.296534522"
Jul 17 13:32:41.284416 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:41.284379    2569 generic.go:356] "Generic (PLEG): container finished" podID="ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9" containerID="3174b55cda303dd6e1f76e4105a761534a463d5e64943b4588f90d037e415d92" exitCode=6
Jul 17 13:32:41.284618 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:41.284466    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z" event={"ID":"ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9","Type":"ContainerDied","Data":"3174b55cda303dd6e1f76e4105a761534a463d5e64943b4588f90d037e415d92"}
Jul 17 13:32:41.284713 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:41.284653    2569 scope.go:117] "RemoveContainer" containerID="3174b55cda303dd6e1f76e4105a761534a463d5e64943b4588f90d037e415d92"
Jul 17 13:32:42.292583 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:42.292547    2569 generic.go:356] "Generic (PLEG): container finished" podID="ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9" containerID="1fa0e51665da1885ea43fb9ad5befe5327ada3553c1fa3695011dfe7bd57d478" exitCode=6
Jul 17 13:32:42.293040 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:42.292604    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z" event={"ID":"ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9","Type":"ContainerDied","Data":"1fa0e51665da1885ea43fb9ad5befe5327ada3553c1fa3695011dfe7bd57d478"}
Jul 17 13:32:42.293040 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:42.292710    2569 scope.go:117] "RemoveContainer" containerID="3174b55cda303dd6e1f76e4105a761534a463d5e64943b4588f90d037e415d92"
Jul 17 13:32:42.293040 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:42.292920    2569 scope.go:117] "RemoveContainer" containerID="1fa0e51665da1885ea43fb9ad5befe5327ada3553c1fa3695011dfe7bd57d478"
Jul 17 13:32:42.293168 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:32:42.293123    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z_odh-ai-gateway-infra(ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z" podUID="ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9"
Jul 17 13:32:43.299171 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:43.299146    2569 scope.go:117] "RemoveContainer" containerID="1fa0e51665da1885ea43fb9ad5befe5327ada3553c1fa3695011dfe7bd57d478"
Jul 17 13:32:43.299538 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:32:43.299316    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z_odh-ai-gateway-infra(ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z" podUID="ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9"
Jul 17 13:32:44.346353 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:44.346322    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"]
Jul 17 13:32:44.346811 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:44.346518    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt" podUID="0231dd62-5202-4852-8853-324f8b8d94c0" containerName="maas-api" containerID="cri-o://016aadb2971e520934d763e221c4d78e91f213b51cd620ee6e28c0fca3a6caf0" gracePeriod=30
Jul 17 13:32:44.604791 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:44.604735    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"
Jul 17 13:32:44.735156 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:44.735129    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkcgn\" (UniqueName: \"kubernetes.io/projected/0231dd62-5202-4852-8853-324f8b8d94c0-kube-api-access-dkcgn\") pod \"0231dd62-5202-4852-8853-324f8b8d94c0\" (UID: \"0231dd62-5202-4852-8853-324f8b8d94c0\") "
Jul 17 13:32:44.735272 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:44.735191    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0231dd62-5202-4852-8853-324f8b8d94c0-maas-api-tls\") pod \"0231dd62-5202-4852-8853-324f8b8d94c0\" (UID: \"0231dd62-5202-4852-8853-324f8b8d94c0\") "
Jul 17 13:32:44.735315 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:44.735296    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0231dd62-5202-4852-8853-324f8b8d94c0-openshift-service-ca\") pod \"0231dd62-5202-4852-8853-324f8b8d94c0\" (UID: \"0231dd62-5202-4852-8853-324f8b8d94c0\") "
Jul 17 13:32:44.735627 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:44.735601    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0231dd62-5202-4852-8853-324f8b8d94c0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0231dd62-5202-4852-8853-324f8b8d94c0" (UID: "0231dd62-5202-4852-8853-324f8b8d94c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:32:44.737314 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:44.737286    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0231dd62-5202-4852-8853-324f8b8d94c0-kube-api-access-dkcgn" (OuterVolumeSpecName: "kube-api-access-dkcgn") pod "0231dd62-5202-4852-8853-324f8b8d94c0" (UID: "0231dd62-5202-4852-8853-324f8b8d94c0"). InnerVolumeSpecName "kube-api-access-dkcgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:32:44.737402 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:44.737285    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0231dd62-5202-4852-8853-324f8b8d94c0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0231dd62-5202-4852-8853-324f8b8d94c0" (UID: "0231dd62-5202-4852-8853-324f8b8d94c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:32:44.836812 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:44.836783    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0231dd62-5202-4852-8853-324f8b8d94c0-openshift-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:32:44.836812 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:44.836813    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkcgn\" (UniqueName: \"kubernetes.io/projected/0231dd62-5202-4852-8853-324f8b8d94c0-kube-api-access-dkcgn\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:32:44.836931 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:44.836824    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0231dd62-5202-4852-8853-324f8b8d94c0-maas-api-tls\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:32:45.310068 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:45.310005    2569 generic.go:356] "Generic (PLEG): container finished" podID="0231dd62-5202-4852-8853-324f8b8d94c0" containerID="016aadb2971e520934d763e221c4d78e91f213b51cd620ee6e28c0fca3a6caf0" exitCode=0
Jul 17 13:32:45.310068 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:45.310055    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt" event={"ID":"0231dd62-5202-4852-8853-324f8b8d94c0","Type":"ContainerDied","Data":"016aadb2971e520934d763e221c4d78e91f213b51cd620ee6e28c0fca3a6caf0"}
Jul 17 13:32:45.310255 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:45.310097    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt" event={"ID":"0231dd62-5202-4852-8853-324f8b8d94c0","Type":"ContainerDied","Data":"827fc19fc3508e00e4f8a1b6420166c4e00e3da312aa9fd66f4bd0ece3a3b40c"}
Jul 17 13:32:45.310255 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:45.310121    2569 scope.go:117] "RemoveContainer" containerID="016aadb2971e520934d763e221c4d78e91f213b51cd620ee6e28c0fca3a6caf0"
Jul 17 13:32:45.310255 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:45.310056    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"
Jul 17 13:32:45.319817 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:45.319796    2569 scope.go:117] "RemoveContainer" containerID="016aadb2971e520934d763e221c4d78e91f213b51cd620ee6e28c0fca3a6caf0"
Jul 17 13:32:45.320104 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:32:45.320083    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"016aadb2971e520934d763e221c4d78e91f213b51cd620ee6e28c0fca3a6caf0\": container with ID starting with 016aadb2971e520934d763e221c4d78e91f213b51cd620ee6e28c0fca3a6caf0 not found: ID does not exist" containerID="016aadb2971e520934d763e221c4d78e91f213b51cd620ee6e28c0fca3a6caf0"
Jul 17 13:32:45.320165 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:45.320113    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"016aadb2971e520934d763e221c4d78e91f213b51cd620ee6e28c0fca3a6caf0"} err="failed to get container status \"016aadb2971e520934d763e221c4d78e91f213b51cd620ee6e28c0fca3a6caf0\": rpc error: code = NotFound desc = could not find container \"016aadb2971e520934d763e221c4d78e91f213b51cd620ee6e28c0fca3a6caf0\": container with ID starting with 016aadb2971e520934d763e221c4d78e91f213b51cd620ee6e28c0fca3a6caf0 not found: ID does not exist"
Jul 17 13:32:45.331979 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:45.331951    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"]
Jul 17 13:32:45.335797 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:45.335770    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e92eab-68dcfc8577-f52qt"]
Jul 17 13:32:45.634053 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:45.633990    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0231dd62-5202-4852-8853-324f8b8d94c0" path="/var/lib/kubelet/pods/0231dd62-5202-4852-8853-324f8b8d94c0/volumes"
Jul 17 13:32:47.629119 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:47.629082    2569 scope.go:117] "RemoveContainer" containerID="692e5b717e42b6c4ce3d94e708bc6a2962fa7276322913c97f9c9d96bfaecdef"
Jul 17 13:32:47.866116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:47.866090    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:32:48.335663 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:48.335634    2569 generic.go:356] "Generic (PLEG): container finished" podID="1b304a07-e0a5-4174-a2bd-771c1f7238de" containerID="0db13d3c563d33c175512ca80bba0c38f694a52fbed18591a86bf8f565992d2b" exitCode=6
Jul 17 13:32:48.335794 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:48.335706    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln" event={"ID":"1b304a07-e0a5-4174-a2bd-771c1f7238de","Type":"ContainerDied","Data":"0db13d3c563d33c175512ca80bba0c38f694a52fbed18591a86bf8f565992d2b"}
Jul 17 13:32:48.335864 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:48.335800    2569 scope.go:117] "RemoveContainer" containerID="692e5b717e42b6c4ce3d94e708bc6a2962fa7276322913c97f9c9d96bfaecdef"
Jul 17 13:32:48.336106 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:48.336084    2569 scope.go:117] "RemoveContainer" containerID="0db13d3c563d33c175512ca80bba0c38f694a52fbed18591a86bf8f565992d2b"
Jul 17 13:32:48.336282 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:32:48.336262    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln_odh-ai-gateway-infra(1b304a07-e0a5-4174-a2bd-771c1f7238de)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln" podUID="1b304a07-e0a5-4174-a2bd-771c1f7238de"
Jul 17 13:32:49.362808 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:49.362768    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln"]
Jul 17 13:32:49.502610 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:49.502589    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln"
Jul 17 13:32:49.673381 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:49.673322    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b304a07-e0a5-4174-a2bd-771c1f7238de-maas-api-service-ca\") pod \"1b304a07-e0a5-4174-a2bd-771c1f7238de\" (UID: \"1b304a07-e0a5-4174-a2bd-771c1f7238de\") "
Jul 17 13:32:49.673684 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:49.673663    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b304a07-e0a5-4174-a2bd-771c1f7238de-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b304a07-e0a5-4174-a2bd-771c1f7238de" (UID: "1b304a07-e0a5-4174-a2bd-771c1f7238de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:32:49.773912 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:49.773889    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b304a07-e0a5-4174-a2bd-771c1f7238de-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:32:50.350083 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:50.350054    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln"
Jul 17 13:32:50.350240 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:50.350095    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln" event={"ID":"1b304a07-e0a5-4174-a2bd-771c1f7238de","Type":"ContainerDied","Data":"ff1471cf138f4476f1e0cc4b0b26364a140a4f30bfac80733be9dc2f05217b53"}
Jul 17 13:32:50.350240 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:50.350164    2569 scope.go:117] "RemoveContainer" containerID="0db13d3c563d33c175512ca80bba0c38f694a52fbed18591a86bf8f565992d2b"
Jul 17 13:32:50.376418 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:50.376390    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln"]
Jul 17 13:32:50.384192 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:50.384165    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vzxln"]
Jul 17 13:32:50.628832 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:50.628774    2569 scope.go:117] "RemoveContainer" containerID="2b3841b4bbcb9d5d3b0c90bff9477a1a71ca97dff46b6dc0951a9ee8e6baefdb"
Jul 17 13:32:51.356668 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:51.356640    2569 generic.go:356] "Generic (PLEG): container finished" podID="7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5" containerID="bca64758f7c492f7176e42b7f20ac6e6f59b2d7d4f2ae422d9236bb736d7ce26" exitCode=6
Jul 17 13:32:51.356815 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:51.356716    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf" event={"ID":"7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5","Type":"ContainerDied","Data":"bca64758f7c492f7176e42b7f20ac6e6f59b2d7d4f2ae422d9236bb736d7ce26"}
Jul 17 13:32:51.356815 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:51.356800    2569 scope.go:117] "RemoveContainer" containerID="2b3841b4bbcb9d5d3b0c90bff9477a1a71ca97dff46b6dc0951a9ee8e6baefdb"
Jul 17 13:32:51.357110 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:51.357075    2569 scope.go:117] "RemoveContainer" containerID="bca64758f7c492f7176e42b7f20ac6e6f59b2d7d4f2ae422d9236bb736d7ce26"
Jul 17 13:32:51.357289 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:32:51.357264    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf_odh-ai-gateway-infra(7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf" podUID="7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5"
Jul 17 13:32:51.633687 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:51.633619    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b304a07-e0a5-4174-a2bd-771c1f7238de" path="/var/lib/kubelet/pods/1b304a07-e0a5-4174-a2bd-771c1f7238de/volumes"
Jul 17 13:32:52.382204 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:52.382149    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf"]
Jul 17 13:32:52.522983 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:52.522960    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf"
Jul 17 13:32:52.593093 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:52.593065    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5-maas-api-service-ca\") pod \"7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5\" (UID: \"7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5\") "
Jul 17 13:32:52.593376 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:52.593355    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5" (UID: "7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:32:52.694451 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:52.694395    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:32:53.371877 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:53.371846    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf" event={"ID":"7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5","Type":"ContainerDied","Data":"bdeb7e6bdcb05c297e1c73cb01684c54edd908c8d61cca152e9ad40eead09d4b"}
Jul 17 13:32:53.372103 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:53.371881    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf"
Jul 17 13:32:53.372103 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:53.371909    2569 scope.go:117] "RemoveContainer" containerID="bca64758f7c492f7176e42b7f20ac6e6f59b2d7d4f2ae422d9236bb736d7ce26"
Jul 17 13:32:53.395030 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:53.394997    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf"]
Jul 17 13:32:53.400673 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:53.400642    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-gszkf"]
Jul 17 13:32:53.633657 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:53.633594    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5" path="/var/lib/kubelet/pods/7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5/volumes"
Jul 17 13:32:55.633924 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:55.633898    2569 scope.go:117] "RemoveContainer" containerID="1fa0e51665da1885ea43fb9ad5befe5327ada3553c1fa3695011dfe7bd57d478"
Jul 17 13:32:56.388087 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:56.388060    2569 generic.go:356] "Generic (PLEG): container finished" podID="ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9" containerID="75e5750b4e592f4ac46ed5e0402dd84fe047d1474f53a17cb76871f4a4a10ad3" exitCode=6
Jul 17 13:32:56.388220 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:56.388140    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z" event={"ID":"ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9","Type":"ContainerDied","Data":"75e5750b4e592f4ac46ed5e0402dd84fe047d1474f53a17cb76871f4a4a10ad3"}
Jul 17 13:32:56.388220 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:56.388206    2569 scope.go:117] "RemoveContainer" containerID="1fa0e51665da1885ea43fb9ad5befe5327ada3553c1fa3695011dfe7bd57d478"
Jul 17 13:32:56.388531 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:56.388505    2569 scope.go:117] "RemoveContainer" containerID="75e5750b4e592f4ac46ed5e0402dd84fe047d1474f53a17cb76871f4a4a10ad3"
Jul 17 13:32:56.388729 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:32:56.388703    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z_odh-ai-gateway-infra(ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z" podUID="ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9"
Jul 17 13:32:57.413003 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:57.412962    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z"]
Jul 17 13:32:57.561431 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:57.561409    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z"
Jul 17 13:32:57.633859 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:57.633834    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9-maas-api-service-ca\") pod \"ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9\" (UID: \"ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9\") "
Jul 17 13:32:57.634241 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:57.634217    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9" (UID: "ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:32:57.735772 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:57.735717    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:32:58.410174 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:58.410141    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z"
Jul 17 13:32:58.410323 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:58.410176    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z" event={"ID":"ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9","Type":"ContainerDied","Data":"a2235bf80134f6f92202766272608e24f2a9c4aa87e6decd0cef1df06f7e49aa"}
Jul 17 13:32:58.410323 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:58.410266    2569 scope.go:117] "RemoveContainer" containerID="75e5750b4e592f4ac46ed5e0402dd84fe047d1474f53a17cb76871f4a4a10ad3"
Jul 17 13:32:58.431442 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:58.431412    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z"]
Jul 17 13:32:58.434744 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:58.434718    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-ztd7z"]
Jul 17 13:32:59.634197 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:32:59.634168    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9" path="/var/lib/kubelet/pods/ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9/volumes"
Jul 17 13:33:00.062360 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.062325    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5c3765-kr65p"]
Jul 17 13:33:00.062820 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.062790    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b304a07-e0a5-4174-a2bd-771c1f7238de" containerName="revoke-keys"
Jul 17 13:33:00.062820 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.062814    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b304a07-e0a5-4174-a2bd-771c1f7238de" containerName="revoke-keys"
Jul 17 13:33:00.062990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.062829    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9" containerName="revoke-keys"
Jul 17 13:33:00.062990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.062838    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9" containerName="revoke-keys"
Jul 17 13:33:00.062990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.062857    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5" containerName="revoke-keys"
Jul 17 13:33:00.062990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.062864    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5" containerName="revoke-keys"
Jul 17 13:33:00.062990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.062883    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5" containerName="revoke-keys"
Jul 17 13:33:00.062990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.062892    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5" containerName="revoke-keys"
Jul 17 13:33:00.062990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.062904    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9" containerName="revoke-keys"
Jul 17 13:33:00.062990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.062917    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9" containerName="revoke-keys"
Jul 17 13:33:00.062990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.062930    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b304a07-e0a5-4174-a2bd-771c1f7238de" containerName="revoke-keys"
Jul 17 13:33:00.062990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.062939    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b304a07-e0a5-4174-a2bd-771c1f7238de" containerName="revoke-keys"
Jul 17 13:33:00.062990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.062949    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0231dd62-5202-4852-8853-324f8b8d94c0" containerName="maas-api"
Jul 17 13:33:00.062990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.062957    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="0231dd62-5202-4852-8853-324f8b8d94c0" containerName="maas-api"
Jul 17 13:33:00.062990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.062966    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5" containerName="revoke-keys"
Jul 17 13:33:00.062990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.062974    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5" containerName="revoke-keys"
Jul 17 13:33:00.063407 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.063060    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b304a07-e0a5-4174-a2bd-771c1f7238de" containerName="revoke-keys"
Jul 17 13:33:00.063407 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.063072    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5" containerName="revoke-keys"
Jul 17 13:33:00.063407 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.063082    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9" containerName="revoke-keys"
Jul 17 13:33:00.063407 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.063090    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9" containerName="revoke-keys"
Jul 17 13:33:00.063407 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.063096    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b304a07-e0a5-4174-a2bd-771c1f7238de" containerName="revoke-keys"
Jul 17 13:33:00.063407 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.063105    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5" containerName="revoke-keys"
Jul 17 13:33:00.063407 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.063115    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="0231dd62-5202-4852-8853-324f8b8d94c0" containerName="maas-api"
Jul 17 13:33:00.063407 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.063123    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9" containerName="revoke-keys"
Jul 17 13:33:00.063407 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.063131    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f2c2f0a-37d4-4a99-a28b-58d0d2bfd3b5" containerName="revoke-keys"
Jul 17 13:33:00.067541 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.067523    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5c3765-kr65p"
Jul 17 13:33:00.070295 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.070248    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:33:00.074062 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.074038    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5c3765-kr65p"]
Jul 17 13:33:00.155717 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.155692    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e342b9-3fac-41c1-8dfa-a0d57b56f2f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-5c3765-kr65p\" (UID: \"30e342b9-3fac-41c1-8dfa-a0d57b56f2f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5c3765-kr65p"
Jul 17 13:33:00.256072 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.256047    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e342b9-3fac-41c1-8dfa-a0d57b56f2f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-5c3765-kr65p\" (UID: \"30e342b9-3fac-41c1-8dfa-a0d57b56f2f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5c3765-kr65p"
Jul 17 13:33:00.256566 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.256550    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e342b9-3fac-41c1-8dfa-a0d57b56f2f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-5c3765-kr65p\" (UID: \"30e342b9-3fac-41c1-8dfa-a0d57b56f2f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5c3765-kr65p"
Jul 17 13:33:00.379718 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.379664    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5c3765-kr65p"
Jul 17 13:33:00.506240 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:00.506211    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5c3765-kr65p"]
Jul 17 13:33:00.508539 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:33:00.508509    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e342b9_3fac_41c1_8dfa_a0d57b56f2f0.slice/crio-cbaeedd98b22fc9d1d9f6d8342275b7cfd17b6a125df9aef44883db7e4a54c77 WatchSource:0}: Error finding container cbaeedd98b22fc9d1d9f6d8342275b7cfd17b6a125df9aef44883db7e4a54c77: Status 404 returned error can't find the container with id cbaeedd98b22fc9d1d9f6d8342275b7cfd17b6a125df9aef44883db7e4a54c77
Jul 17 13:33:01.428090 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:01.428019    2569 generic.go:356] "Generic (PLEG): container finished" podID="30e342b9-3fac-41c1-8dfa-a0d57b56f2f0" containerID="7c64adbf5dd8618e4dfe05bff1bf124ca02075d72224574fdf8394fd10eba91d" exitCode=0
Jul 17 13:33:01.428423 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:01.428089    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5c3765-kr65p" event={"ID":"30e342b9-3fac-41c1-8dfa-a0d57b56f2f0","Type":"ContainerDied","Data":"7c64adbf5dd8618e4dfe05bff1bf124ca02075d72224574fdf8394fd10eba91d"}
Jul 17 13:33:01.428423 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:01.428141    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5c3765-kr65p" event={"ID":"30e342b9-3fac-41c1-8dfa-a0d57b56f2f0","Type":"ContainerStarted","Data":"cbaeedd98b22fc9d1d9f6d8342275b7cfd17b6a125df9aef44883db7e4a54c77"}
Jul 17 13:33:02.564655 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:02.564631    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5c3765-kr65p"
Jul 17 13:33:02.672107 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:02.672076    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e342b9-3fac-41c1-8dfa-a0d57b56f2f0-maas-api-service-ca\") pod \"30e342b9-3fac-41c1-8dfa-a0d57b56f2f0\" (UID: \"30e342b9-3fac-41c1-8dfa-a0d57b56f2f0\") "
Jul 17 13:33:02.672394 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:02.672375    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e342b9-3fac-41c1-8dfa-a0d57b56f2f0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "30e342b9-3fac-41c1-8dfa-a0d57b56f2f0" (UID: "30e342b9-3fac-41c1-8dfa-a0d57b56f2f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:33:02.772901 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:02.772880    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e342b9-3fac-41c1-8dfa-a0d57b56f2f0-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:33:03.439390 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:03.439362    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5c3765-kr65p"
Jul 17 13:33:03.439522 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:03.439394    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5c3765-kr65p" event={"ID":"30e342b9-3fac-41c1-8dfa-a0d57b56f2f0","Type":"ContainerDied","Data":"cbaeedd98b22fc9d1d9f6d8342275b7cfd17b6a125df9aef44883db7e4a54c77"}
Jul 17 13:33:03.439522 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:03.439450    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbaeedd98b22fc9d1d9f6d8342275b7cfd17b6a125df9aef44883db7e4a54c77"
Jul 17 13:33:10.578172 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:10.578138    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"]
Jul 17 13:33:10.578599 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:10.578333    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq" podUID="57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3" containerName="maas-api" containerID="cri-o://b8a306a676bd246e164cce1169d935247fa3bb56b195b4f9445e2f1f7c085249" gracePeriod=30
Jul 17 13:33:10.822647 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:10.822622    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"
Jul 17 13:33:10.937986 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:10.937929    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-maas-api-tls\") pod \"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3\" (UID: \"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3\") "
Jul 17 13:33:10.938091 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:10.937989    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-openshift-service-ca\") pod \"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3\" (UID: \"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3\") "
Jul 17 13:33:10.938091 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:10.938044    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg4vp\" (UniqueName: \"kubernetes.io/projected/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-kube-api-access-xg4vp\") pod \"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3\" (UID: \"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3\") "
Jul 17 13:33:10.938361 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:10.938334    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3" (UID: "57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:33:10.940009 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:10.939975    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3" (UID: "57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:33:10.940107 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:10.940084    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-kube-api-access-xg4vp" (OuterVolumeSpecName: "kube-api-access-xg4vp") pod "57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3" (UID: "57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3"). InnerVolumeSpecName "kube-api-access-xg4vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:33:11.039163 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:11.039141    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-maas-api-tls\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:33:11.039163 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:11.039160    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-openshift-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:33:11.039293 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:11.039170    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg4vp\" (UniqueName: \"kubernetes.io/projected/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3-kube-api-access-xg4vp\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:33:11.478361 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:11.478327    2569 generic.go:356] "Generic (PLEG): container finished" podID="57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3" containerID="b8a306a676bd246e164cce1169d935247fa3bb56b195b4f9445e2f1f7c085249" exitCode=0
Jul 17 13:33:11.478499 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:11.478386    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"
Jul 17 13:33:11.478499 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:11.478414    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq" event={"ID":"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3","Type":"ContainerDied","Data":"b8a306a676bd246e164cce1169d935247fa3bb56b195b4f9445e2f1f7c085249"}
Jul 17 13:33:11.478499 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:11.478494    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq" event={"ID":"57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3","Type":"ContainerDied","Data":"e88a67bf12108072695240a6fd245698ef839585b819263c6574f9f5ccbc7063"}
Jul 17 13:33:11.478609 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:11.478523    2569 scope.go:117] "RemoveContainer" containerID="b8a306a676bd246e164cce1169d935247fa3bb56b195b4f9445e2f1f7c085249"
Jul 17 13:33:11.487877 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:11.487851    2569 scope.go:117] "RemoveContainer" containerID="b8a306a676bd246e164cce1169d935247fa3bb56b195b4f9445e2f1f7c085249"
Jul 17 13:33:11.488158 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:33:11.488137    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a306a676bd246e164cce1169d935247fa3bb56b195b4f9445e2f1f7c085249\": container with ID starting with b8a306a676bd246e164cce1169d935247fa3bb56b195b4f9445e2f1f7c085249 not found: ID does not exist" containerID="b8a306a676bd246e164cce1169d935247fa3bb56b195b4f9445e2f1f7c085249"
Jul 17 13:33:11.488203 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:11.488168    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a306a676bd246e164cce1169d935247fa3bb56b195b4f9445e2f1f7c085249"} err="failed to get container status \"b8a306a676bd246e164cce1169d935247fa3bb56b195b4f9445e2f1f7c085249\": rpc error: code = NotFound desc = could not find container \"b8a306a676bd246e164cce1169d935247fa3bb56b195b4f9445e2f1f7c085249\": container with ID starting with b8a306a676bd246e164cce1169d935247fa3bb56b195b4f9445e2f1f7c085249 not found: ID does not exist"
Jul 17 13:33:11.500865 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:11.500841    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"]
Jul 17 13:33:11.504544 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:11.504520    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-5c3765-7465ff4bd9-qwmkq"]
Jul 17 13:33:11.634100 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:11.634075    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3" path="/var/lib/kubelet/pods/57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3/volumes"
Jul 17 13:33:13.664093 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:13.664054    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-mx54h"]
Jul 17 13:33:21.853891 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:21.853848    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv"]
Jul 17 13:33:21.854487 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:21.854347    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30e342b9-3fac-41c1-8dfa-a0d57b56f2f0" containerName="revoke-keys"
Jul 17 13:33:21.854487 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:21.854367    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="30e342b9-3fac-41c1-8dfa-a0d57b56f2f0" containerName="revoke-keys"
Jul 17 13:33:21.854487 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:21.854380    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9" containerName="revoke-keys"
Jul 17 13:33:21.854487 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:21.854389    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded0b8dc-92ef-40ce-8925-cf6ac2c1fdb9" containerName="revoke-keys"
Jul 17 13:33:21.854487 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:21.854405    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b304a07-e0a5-4174-a2bd-771c1f7238de" containerName="revoke-keys"
Jul 17 13:33:21.854487 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:21.854414    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b304a07-e0a5-4174-a2bd-771c1f7238de" containerName="revoke-keys"
Jul 17 13:33:21.854487 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:21.854435    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3" containerName="maas-api"
Jul 17 13:33:21.854487 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:21.854443    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3" containerName="maas-api"
Jul 17 13:33:21.854957 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:21.854520    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="57eeaded-3b80-4b86-afa5-fa0fb8f7f0c3" containerName="maas-api"
Jul 17 13:33:21.854957 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:21.854536    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b304a07-e0a5-4174-a2bd-771c1f7238de" containerName="revoke-keys"
Jul 17 13:33:21.854957 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:21.854549    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="30e342b9-3fac-41c1-8dfa-a0d57b56f2f0" containerName="revoke-keys"
Jul 17 13:33:21.859179 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:21.859155    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv"
Jul 17 13:33:21.862187 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:21.862166    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:33:21.862330 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:21.862165    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:33:21.865426 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:21.865400    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv"]
Jul 17 13:33:21.929950 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:21.929925    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e5d7cb-f678-411b-b88d-1644a9f46bf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv\" (UID: \"f8e5d7cb-f678-411b-b88d-1644a9f46bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv"
Jul 17 13:33:22.030892 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:22.030853    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e5d7cb-f678-411b-b88d-1644a9f46bf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv\" (UID: \"f8e5d7cb-f678-411b-b88d-1644a9f46bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv"
Jul 17 13:33:22.031429 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:22.031410    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e5d7cb-f678-411b-b88d-1644a9f46bf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv\" (UID: \"f8e5d7cb-f678-411b-b88d-1644a9f46bf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv"
Jul 17 13:33:22.172022 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:22.171961    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv"
Jul 17 13:33:22.300933 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:22.300877    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv"]
Jul 17 13:33:22.303046 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:33:22.303019    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e5d7cb_f678_411b_b88d_1644a9f46bf1.slice/crio-9bf327edb68bd1ea7ef59e838296c513b976b194878ddcd1c3b6989a7439940f WatchSource:0}: Error finding container 9bf327edb68bd1ea7ef59e838296c513b976b194878ddcd1c3b6989a7439940f: Status 404 returned error can't find the container with id 9bf327edb68bd1ea7ef59e838296c513b976b194878ddcd1c3b6989a7439940f
Jul 17 13:33:22.533478 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:22.533448    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv" event={"ID":"f8e5d7cb-f678-411b-b88d-1644a9f46bf1","Type":"ContainerStarted","Data":"9bf327edb68bd1ea7ef59e838296c513b976b194878ddcd1c3b6989a7439940f"}
Jul 17 13:33:23.540325 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:23.540286    2569 generic.go:356] "Generic (PLEG): container finished" podID="f8e5d7cb-f678-411b-b88d-1644a9f46bf1" containerID="9a0ef0435313d6d5143abd62a801c36b475bf9634daee9262e992556093cf81d" exitCode=6
Jul 17 13:33:23.540713 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:23.540375    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv" event={"ID":"f8e5d7cb-f678-411b-b88d-1644a9f46bf1","Type":"ContainerDied","Data":"9a0ef0435313d6d5143abd62a801c36b475bf9634daee9262e992556093cf81d"}
Jul 17 13:33:23.540713 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:23.540551    2569 scope.go:117] "RemoveContainer" containerID="9a0ef0435313d6d5143abd62a801c36b475bf9634daee9262e992556093cf81d"
Jul 17 13:33:24.547134 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:24.547100    2569 generic.go:356] "Generic (PLEG): container finished" podID="f8e5d7cb-f678-411b-b88d-1644a9f46bf1" containerID="d615a320e780a8a114af317a79395a7752ed07be9214a05217471c3a42b14b15" exitCode=6
Jul 17 13:33:24.547546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:24.547133    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv" event={"ID":"f8e5d7cb-f678-411b-b88d-1644a9f46bf1","Type":"ContainerDied","Data":"d615a320e780a8a114af317a79395a7752ed07be9214a05217471c3a42b14b15"}
Jul 17 13:33:24.547546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:24.547205    2569 scope.go:117] "RemoveContainer" containerID="9a0ef0435313d6d5143abd62a801c36b475bf9634daee9262e992556093cf81d"
Jul 17 13:33:24.547546 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:24.547371    2569 scope.go:117] "RemoveContainer" containerID="d615a320e780a8a114af317a79395a7752ed07be9214a05217471c3a42b14b15"
Jul 17 13:33:24.547709 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:33:24.547559    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv_odh-ai-gateway-infra(f8e5d7cb-f678-411b-b88d-1644a9f46bf1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv" podUID="f8e5d7cb-f678-411b-b88d-1644a9f46bf1"
Jul 17 13:33:25.554174 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:25.554152    2569 scope.go:117] "RemoveContainer" containerID="d615a320e780a8a114af317a79395a7752ed07be9214a05217471c3a42b14b15"
Jul 17 13:33:25.554508 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:33:25.554312    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv_odh-ai-gateway-infra(f8e5d7cb-f678-411b-b88d-1644a9f46bf1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv" podUID="f8e5d7cb-f678-411b-b88d-1644a9f46bf1"
Jul 17 13:33:29.330128 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:29.330090    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh"]
Jul 17 13:33:29.334408 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:29.334383    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh"
Jul 17 13:33:29.343353 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:29.343326    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh"]
Jul 17 13:33:29.485584 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:29.485558    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16679337-25ad-4a0f-bc6a-db2f1f7f643d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh\" (UID: \"16679337-25ad-4a0f-bc6a-db2f1f7f643d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh"
Jul 17 13:33:29.585945 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:29.585886    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16679337-25ad-4a0f-bc6a-db2f1f7f643d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh\" (UID: \"16679337-25ad-4a0f-bc6a-db2f1f7f643d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh"
Jul 17 13:33:29.586484 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:29.586463    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16679337-25ad-4a0f-bc6a-db2f1f7f643d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh\" (UID: \"16679337-25ad-4a0f-bc6a-db2f1f7f643d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh"
Jul 17 13:33:29.645537 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:29.645517    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh"
Jul 17 13:33:29.771241 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:29.771215    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh"]
Jul 17 13:33:29.773651 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:33:29.773617    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16679337_25ad_4a0f_bc6a_db2f1f7f643d.slice/crio-e662ed0ae471b93245e952ba18ef47af2277ee514541bef62f870a7b8b1dbcfb WatchSource:0}: Error finding container e662ed0ae471b93245e952ba18ef47af2277ee514541bef62f870a7b8b1dbcfb: Status 404 returned error can't find the container with id e662ed0ae471b93245e952ba18ef47af2277ee514541bef62f870a7b8b1dbcfb
Jul 17 13:33:30.578362 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:30.578297    2569 generic.go:356] "Generic (PLEG): container finished" podID="16679337-25ad-4a0f-bc6a-db2f1f7f643d" containerID="1796fc8832aea52b4cf3842369a244d7099b223f35bc456de7017daa51112049" exitCode=6
Jul 17 13:33:30.578362 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:30.578338    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh" event={"ID":"16679337-25ad-4a0f-bc6a-db2f1f7f643d","Type":"ContainerDied","Data":"1796fc8832aea52b4cf3842369a244d7099b223f35bc456de7017daa51112049"}
Jul 17 13:33:30.578703 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:30.578380    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh" event={"ID":"16679337-25ad-4a0f-bc6a-db2f1f7f643d","Type":"ContainerStarted","Data":"e662ed0ae471b93245e952ba18ef47af2277ee514541bef62f870a7b8b1dbcfb"}
Jul 17 13:33:30.578703 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:30.578523    2569 scope.go:117] "RemoveContainer" containerID="1796fc8832aea52b4cf3842369a244d7099b223f35bc456de7017daa51112049"
Jul 17 13:33:31.584702 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:31.584673    2569 generic.go:356] "Generic (PLEG): container finished" podID="16679337-25ad-4a0f-bc6a-db2f1f7f643d" containerID="ee508a0f84b591e01d9b846bcddb13dec55c3b740d06e559b15b2b2e0b842da7" exitCode=6
Jul 17 13:33:31.585103 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:31.584715    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh" event={"ID":"16679337-25ad-4a0f-bc6a-db2f1f7f643d","Type":"ContainerDied","Data":"ee508a0f84b591e01d9b846bcddb13dec55c3b740d06e559b15b2b2e0b842da7"}
Jul 17 13:33:31.585103 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:31.584788    2569 scope.go:117] "RemoveContainer" containerID="1796fc8832aea52b4cf3842369a244d7099b223f35bc456de7017daa51112049"
Jul 17 13:33:31.585103 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:31.584935    2569 scope.go:117] "RemoveContainer" containerID="ee508a0f84b591e01d9b846bcddb13dec55c3b740d06e559b15b2b2e0b842da7"
Jul 17 13:33:31.585251 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:33:31.585118    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh_odh-ai-gateway-infra(16679337-25ad-4a0f-bc6a-db2f1f7f643d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh" podUID="16679337-25ad-4a0f-bc6a-db2f1f7f643d"
Jul 17 13:33:32.591438 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:32.591416    2569 scope.go:117] "RemoveContainer" containerID="ee508a0f84b591e01d9b846bcddb13dec55c3b740d06e559b15b2b2e0b842da7"
Jul 17 13:33:32.591800 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:33:32.591572    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh_odh-ai-gateway-infra(16679337-25ad-4a0f-bc6a-db2f1f7f643d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh" podUID="16679337-25ad-4a0f-bc6a-db2f1f7f643d"
Jul 17 13:33:32.639719 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:32.639672    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d8fb4667b-ccn8z_1272b7db-2ffc-4f97-b60a-693a9b5749d6/manager/0.log"
Jul 17 13:33:32.649298 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:32.649277    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d8fb4667b-ccn8z_1272b7db-2ffc-4f97-b60a-693a9b5749d6/copy-manifests/0.log"
Jul 17 13:33:33.188378 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:33.188345    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2"]
Jul 17 13:33:33.193069 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:33.193047    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2"
Jul 17 13:33:33.199363 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:33.199337    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2"]
Jul 17 13:33:33.313116 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:33.313090    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94949073-5836-48f3-8abb-9e45b94ceca0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2\" (UID: \"94949073-5836-48f3-8abb-9e45b94ceca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2"
Jul 17 13:33:33.347917 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:33.347894    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738235-fk7wd_8c391ea4-c0bd-4ab0-bc86-2fa99f79ce50/cleanup/0.log"
Jul 17 13:33:33.414385 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:33.414365    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94949073-5836-48f3-8abb-9e45b94ceca0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2\" (UID: \"94949073-5836-48f3-8abb-9e45b94ceca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2"
Jul 17 13:33:33.414982 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:33.414962    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94949073-5836-48f3-8abb-9e45b94ceca0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2\" (UID: \"94949073-5836-48f3-8abb-9e45b94ceca0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2"
Jul 17 13:33:33.506514 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:33.506489    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2"
Jul 17 13:33:33.641521 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:33.641498    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2"]
Jul 17 13:33:33.643721 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:33:33.643694    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94949073_5836_48f3_8abb_9e45b94ceca0.slice/crio-24fa26faed8c2569f58d99019474c8fcdd4a2932553b0f449101c23c16ce0b85 WatchSource:0}: Error finding container 24fa26faed8c2569f58d99019474c8fcdd4a2932553b0f449101c23c16ce0b85: Status 404 returned error can't find the container with id 24fa26faed8c2569f58d99019474c8fcdd4a2932553b0f449101c23c16ce0b85
Jul 17 13:33:33.805998 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:33.805944    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-6fa40d-9bdsp_1c98eb2e-3e3f-4fd6-8701-8659bb1994cd/revoke-keys/0.log"
Jul 17 13:33:33.921998 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:33.921973    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh_16679337-25ad-4a0f-bc6a-db2f1f7f643d/revoke-keys/1.log"
Jul 17 13:33:34.157464 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:34.157406    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-5dd9aa-vqsfm_b9f65aee-9ffc-4c0d-a44e-2f9d7a13f502/revoke-keys/0.log"
Jul 17 13:33:34.273732 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:34.273710    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv_f8e5d7cb-f678-411b-b88d-1644a9f46bf1/revoke-keys/1.log"
Jul 17 13:33:34.516366 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:34.516348    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-cae85d5e-8t4sl_581313ce-1e8a-4cdd-8e64-e12bc5ce6e31/revoke-keys/0.log"
Jul 17 13:33:34.602720 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:34.602688    2569 generic.go:356] "Generic (PLEG): container finished" podID="94949073-5836-48f3-8abb-9e45b94ceca0" containerID="02279b5c1ed6015751cac76f2f596f1ab0709cad8095d4604a87499494c2fb5e" exitCode=6
Jul 17 13:33:34.602853 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:34.602788    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2" event={"ID":"94949073-5836-48f3-8abb-9e45b94ceca0","Type":"ContainerDied","Data":"02279b5c1ed6015751cac76f2f596f1ab0709cad8095d4604a87499494c2fb5e"}
Jul 17 13:33:34.602853 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:34.602836    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2" event={"ID":"94949073-5836-48f3-8abb-9e45b94ceca0","Type":"ContainerStarted","Data":"24fa26faed8c2569f58d99019474c8fcdd4a2932553b0f449101c23c16ce0b85"}
Jul 17 13:33:34.602930 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:34.602910    2569 scope.go:117] "RemoveContainer" containerID="02279b5c1ed6015751cac76f2f596f1ab0709cad8095d4604a87499494c2fb5e"
Jul 17 13:33:34.629060 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:34.629032    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-e92eab-69sdj_a99e479e-60c6-4bb2-b497-f744efd9e071/revoke-keys/0.log"
Jul 17 13:33:34.749207 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:34.749184    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-5c3765-kr65p_30e342b9-3fac-41c1-8dfa-a0d57b56f2f0/revoke-keys/0.log"
Jul 17 13:33:35.609885 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:35.609857    2569 generic.go:356] "Generic (PLEG): container finished" podID="94949073-5836-48f3-8abb-9e45b94ceca0" containerID="ba4bb0be937a0c6fb2ec18be21d1fce47e849e92f255c552be27b480ff15952f" exitCode=6
Jul 17 13:33:35.610036 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:35.609911    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2" event={"ID":"94949073-5836-48f3-8abb-9e45b94ceca0","Type":"ContainerDied","Data":"ba4bb0be937a0c6fb2ec18be21d1fce47e849e92f255c552be27b480ff15952f"}
Jul 17 13:33:35.610036 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:35.609977    2569 scope.go:117] "RemoveContainer" containerID="02279b5c1ed6015751cac76f2f596f1ab0709cad8095d4604a87499494c2fb5e"
Jul 17 13:33:35.610150 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:35.610119    2569 scope.go:117] "RemoveContainer" containerID="ba4bb0be937a0c6fb2ec18be21d1fce47e849e92f255c552be27b480ff15952f"
Jul 17 13:33:35.610327 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:33:35.610301    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2_odh-ai-gateway-infra(94949073-5836-48f3-8abb-9e45b94ceca0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2" podUID="94949073-5836-48f3-8abb-9e45b94ceca0"
Jul 17 13:33:36.615835 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:36.615803    2569 scope.go:117] "RemoveContainer" containerID="ba4bb0be937a0c6fb2ec18be21d1fce47e849e92f255c552be27b480ff15952f"
Jul 17 13:33:36.616195 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:33:36.616009    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2_odh-ai-gateway-infra(94949073-5836-48f3-8abb-9e45b94ceca0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2" podUID="94949073-5836-48f3-8abb-9e45b94ceca0"
Jul 17 13:33:36.924428 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:36.924353    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-mx54h_12782b8e-49c1-4930-b60e-d470c50b9bef/limitador/0.log"
Jul 17 13:33:37.515911 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:37.515883    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wh79n_4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb/discovery/0.log"
Jul 17 13:33:38.628236 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:38.628210    2569 scope.go:117] "RemoveContainer" containerID="d615a320e780a8a114af317a79395a7752ed07be9214a05217471c3a42b14b15"
Jul 17 13:33:38.918425 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:38.918376    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbtg9tj_179bc674-34f1-4cec-94b5-e6f3b48fda3f/main/0.log"
Jul 17 13:33:39.634638 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:39.634607    2569 generic.go:356] "Generic (PLEG): container finished" podID="f8e5d7cb-f678-411b-b88d-1644a9f46bf1" containerID="39e7f57b38e7871ebf94adb5be96b40de929eadc9ebc3a957a695b5d14253c52" exitCode=6
Jul 17 13:33:39.635038 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:39.634662    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv" event={"ID":"f8e5d7cb-f678-411b-b88d-1644a9f46bf1","Type":"ContainerDied","Data":"39e7f57b38e7871ebf94adb5be96b40de929eadc9ebc3a957a695b5d14253c52"}
Jul 17 13:33:39.635038 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:39.634715    2569 scope.go:117] "RemoveContainer" containerID="d615a320e780a8a114af317a79395a7752ed07be9214a05217471c3a42b14b15"
Jul 17 13:33:39.635038 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:39.634961    2569 scope.go:117] "RemoveContainer" containerID="39e7f57b38e7871ebf94adb5be96b40de929eadc9ebc3a957a695b5d14253c52"
Jul 17 13:33:39.635175 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:33:39.635156    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv_odh-ai-gateway-infra(f8e5d7cb-f678-411b-b88d-1644a9f46bf1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv" podUID="f8e5d7cb-f678-411b-b88d-1644a9f46bf1"
Jul 17 13:33:40.660456 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:40.660430    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv"]
Jul 17 13:33:40.790994 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:40.790969    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv"
Jul 17 13:33:40.872090 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:40.872060    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e5d7cb-f678-411b-b88d-1644a9f46bf1-maas-api-service-ca\") pod \"f8e5d7cb-f678-411b-b88d-1644a9f46bf1\" (UID: \"f8e5d7cb-f678-411b-b88d-1644a9f46bf1\") "
Jul 17 13:33:40.872385 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:40.872361    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e5d7cb-f678-411b-b88d-1644a9f46bf1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8e5d7cb-f678-411b-b88d-1644a9f46bf1" (UID: "f8e5d7cb-f678-411b-b88d-1644a9f46bf1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:33:40.973621 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:40.973555    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e5d7cb-f678-411b-b88d-1644a9f46bf1-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:33:41.647501 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:41.647477    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv"
Jul 17 13:33:41.647648 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:41.647500    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv" event={"ID":"f8e5d7cb-f678-411b-b88d-1644a9f46bf1","Type":"ContainerDied","Data":"9bf327edb68bd1ea7ef59e838296c513b976b194878ddcd1c3b6989a7439940f"}
Jul 17 13:33:41.647648 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:41.647552    2569 scope.go:117] "RemoveContainer" containerID="39e7f57b38e7871ebf94adb5be96b40de929eadc9ebc3a957a695b5d14253c52"
Jul 17 13:33:41.663734 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:41.663710    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv"]
Jul 17 13:33:41.667940 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:41.667920    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-bdkxv"]
Jul 17 13:33:43.634457 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:43.634419    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8e5d7cb-f678-411b-b88d-1644a9f46bf1" path="/var/lib/kubelet/pods/f8e5d7cb-f678-411b-b88d-1644a9f46bf1/volumes"
Jul 17 13:33:46.628460 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:46.628431    2569 scope.go:117] "RemoveContainer" containerID="ee508a0f84b591e01d9b846bcddb13dec55c3b740d06e559b15b2b2e0b842da7"
Jul 17 13:33:47.628983 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:47.628916    2569 scope.go:117] "RemoveContainer" containerID="ba4bb0be937a0c6fb2ec18be21d1fce47e849e92f255c552be27b480ff15952f"
Jul 17 13:33:47.677051 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:47.677027    2569 generic.go:356] "Generic (PLEG): container finished" podID="16679337-25ad-4a0f-bc6a-db2f1f7f643d" containerID="ce9f99dda501c070854f0f675fd4a13a2048df878ff135ad030f5027e6088386" exitCode=6
Jul 17 13:33:47.677133 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:47.677069    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh" event={"ID":"16679337-25ad-4a0f-bc6a-db2f1f7f643d","Type":"ContainerDied","Data":"ce9f99dda501c070854f0f675fd4a13a2048df878ff135ad030f5027e6088386"}
Jul 17 13:33:47.677133 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:47.677112    2569 scope.go:117] "RemoveContainer" containerID="ee508a0f84b591e01d9b846bcddb13dec55c3b740d06e559b15b2b2e0b842da7"
Jul 17 13:33:47.677343 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:47.677327    2569 scope.go:117] "RemoveContainer" containerID="ce9f99dda501c070854f0f675fd4a13a2048df878ff135ad030f5027e6088386"
Jul 17 13:33:47.677496 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:33:47.677477    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh_odh-ai-gateway-infra(16679337-25ad-4a0f-bc6a-db2f1f7f643d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh" podUID="16679337-25ad-4a0f-bc6a-db2f1f7f643d"
Jul 17 13:33:47.812495 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:47.812475    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nw9sw_a4b940c7-3fe3-4290-b253-efde9655c166/global-pull-secret-syncer/0.log"
Jul 17 13:33:47.944413 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:47.944362    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c2c46_a9916c7a-cfa6-4b17-bea4-0642e4dce23e/konnectivity-agent/0.log"
Jul 17 13:33:48.030044 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:48.030024    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-210.ec2.internal_bc30aca19fb5f39f59f0d9bb7d486e95/haproxy/0.log"
Jul 17 13:33:48.685157 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:48.685131    2569 generic.go:356] "Generic (PLEG): container finished" podID="94949073-5836-48f3-8abb-9e45b94ceca0" containerID="4ba375845397c2cf2b077804ff99142a500bac8066edde8f2ea61f30bdf501b2" exitCode=6
Jul 17 13:33:48.685496 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:48.685205    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2" event={"ID":"94949073-5836-48f3-8abb-9e45b94ceca0","Type":"ContainerDied","Data":"4ba375845397c2cf2b077804ff99142a500bac8066edde8f2ea61f30bdf501b2"}
Jul 17 13:33:48.685496 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:48.685265    2569 scope.go:117] "RemoveContainer" containerID="ba4bb0be937a0c6fb2ec18be21d1fce47e849e92f255c552be27b480ff15952f"
Jul 17 13:33:48.685496 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:48.685491    2569 scope.go:117] "RemoveContainer" containerID="4ba375845397c2cf2b077804ff99142a500bac8066edde8f2ea61f30bdf501b2"
Jul 17 13:33:48.685724 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:33:48.685702    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2_odh-ai-gateway-infra(94949073-5836-48f3-8abb-9e45b94ceca0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2" podUID="94949073-5836-48f3-8abb-9e45b94ceca0"
Jul 17 13:33:48.706131 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:48.706107    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh"]
Jul 17 13:33:48.837407 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:48.837385    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh"
Jul 17 13:33:48.933188 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:48.933161    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16679337-25ad-4a0f-bc6a-db2f1f7f643d-maas-api-service-ca\") pod \"16679337-25ad-4a0f-bc6a-db2f1f7f643d\" (UID: \"16679337-25ad-4a0f-bc6a-db2f1f7f643d\") "
Jul 17 13:33:48.933480 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:48.933458    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16679337-25ad-4a0f-bc6a-db2f1f7f643d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "16679337-25ad-4a0f-bc6a-db2f1f7f643d" (UID: "16679337-25ad-4a0f-bc6a-db2f1f7f643d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:33:49.034481 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:49.034459    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16679337-25ad-4a0f-bc6a-db2f1f7f643d-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:33:49.691691 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:49.691666    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh"
Jul 17 13:33:49.692090 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:49.691665    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh" event={"ID":"16679337-25ad-4a0f-bc6a-db2f1f7f643d","Type":"ContainerDied","Data":"e662ed0ae471b93245e952ba18ef47af2277ee514541bef62f870a7b8b1dbcfb"}
Jul 17 13:33:49.692090 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:49.691816    2569 scope.go:117] "RemoveContainer" containerID="ce9f99dda501c070854f0f675fd4a13a2048df878ff135ad030f5027e6088386"
Jul 17 13:33:49.708037 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:49.708015    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh"]
Jul 17 13:33:49.713341 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:49.713313    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2"]
Jul 17 13:33:49.716007 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:49.715984    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lm2hh"]
Jul 17 13:33:49.848384 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:49.848359    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2"
Jul 17 13:33:49.939332 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:49.939271    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94949073-5836-48f3-8abb-9e45b94ceca0-maas-api-service-ca\") pod \"94949073-5836-48f3-8abb-9e45b94ceca0\" (UID: \"94949073-5836-48f3-8abb-9e45b94ceca0\") "
Jul 17 13:33:49.939572 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:49.939551    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94949073-5836-48f3-8abb-9e45b94ceca0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94949073-5836-48f3-8abb-9e45b94ceca0" (UID: "94949073-5836-48f3-8abb-9e45b94ceca0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:33:50.040277 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:50.040253    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94949073-5836-48f3-8abb-9e45b94ceca0-maas-api-service-ca\") on node \"ip-10-0-131-210.ec2.internal\" DevicePath \"\""
Jul 17 13:33:50.701558 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:50.701521    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2" event={"ID":"94949073-5836-48f3-8abb-9e45b94ceca0","Type":"ContainerDied","Data":"24fa26faed8c2569f58d99019474c8fcdd4a2932553b0f449101c23c16ce0b85"}
Jul 17 13:33:50.701980 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:50.701581    2569 scope.go:117] "RemoveContainer" containerID="4ba375845397c2cf2b077804ff99142a500bac8066edde8f2ea61f30bdf501b2"
Jul 17 13:33:50.701980 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:50.701542    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2"
Jul 17 13:33:50.725289 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:50.725250    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2"]
Jul 17 13:33:50.733001 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:50.732979    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-nb2z2"]
Jul 17 13:33:51.634602 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:51.634566    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16679337-25ad-4a0f-bc6a-db2f1f7f643d" path="/var/lib/kubelet/pods/16679337-25ad-4a0f-bc6a-db2f1f7f643d/volumes"
Jul 17 13:33:51.634962 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:51.634944    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94949073-5836-48f3-8abb-9e45b94ceca0" path="/var/lib/kubelet/pods/94949073-5836-48f3-8abb-9e45b94ceca0/volumes"
Jul 17 13:33:53.851603 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:53.851579    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-mx54h_12782b8e-49c1-4930-b60e-d470c50b9bef/limitador/0.log"
Jul 17 13:33:55.790245 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:55.790222    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-85cbb9955f-4hzpd_fcb3eb8a-314a-44d7-a076-5267528a012c/metrics-server/0.log"
Jul 17 13:33:55.866969 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:55.866911    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2j4mj_bcff7d42-41a1-45d3-b58a-e11ac6c40173/node-exporter/0.log"
Jul 17 13:33:55.894839 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:55.894811    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2j4mj_bcff7d42-41a1-45d3-b58a-e11ac6c40173/kube-rbac-proxy/0.log"
Jul 17 13:33:55.924881 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:55.924864    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2j4mj_bcff7d42-41a1-45d3-b58a-e11ac6c40173/init-textfile/0.log"
Jul 17 13:33:56.111701 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:56.111677    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-ntjbj_59ba81f3-2110-48a1-9079-90e55119c423/kube-rbac-proxy-main/0.log"
Jul 17 13:33:56.133549 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:56.133497    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-ntjbj_59ba81f3-2110-48a1-9079-90e55119c423/kube-rbac-proxy-self/0.log"
Jul 17 13:33:56.156212 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:56.156193    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-ntjbj_59ba81f3-2110-48a1-9079-90e55119c423/openshift-state-metrics/0.log"
Jul 17 13:33:56.378514 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:56.378497    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-9cvww_1810c267-db7d-4336-b21c-cde97c4de6e1/prometheus-operator/0.log"
Jul 17 13:33:56.400528 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:56.400479    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-9cvww_1810c267-db7d-4336-b21c-cde97c4de6e1/kube-rbac-proxy/0.log"
Jul 17 13:33:56.528221 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:56.528201    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-587b64d868-wpsz7_493d99ab-0d82-456d-b003-964aec113cc4/telemeter-client/0.log"
Jul 17 13:33:56.568940 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:56.568920    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-587b64d868-wpsz7_493d99ab-0d82-456d-b003-964aec113cc4/reload/0.log"
Jul 17 13:33:56.602430 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:56.602410    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-587b64d868-wpsz7_493d99ab-0d82-456d-b003-964aec113cc4/kube-rbac-proxy/0.log"
Jul 17 13:33:58.251202 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251104    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"]
Jul 17 13:33:58.251574 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251463    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94949073-5836-48f3-8abb-9e45b94ceca0" containerName="revoke-keys"
Jul 17 13:33:58.251574 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251475    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="94949073-5836-48f3-8abb-9e45b94ceca0" containerName="revoke-keys"
Jul 17 13:33:58.251574 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251491    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e5d7cb-f678-411b-b88d-1644a9f46bf1" containerName="revoke-keys"
Jul 17 13:33:58.251574 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251497    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e5d7cb-f678-411b-b88d-1644a9f46bf1" containerName="revoke-keys"
Jul 17 13:33:58.251574 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251505    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16679337-25ad-4a0f-bc6a-db2f1f7f643d" containerName="revoke-keys"
Jul 17 13:33:58.251574 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251511    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="16679337-25ad-4a0f-bc6a-db2f1f7f643d" containerName="revoke-keys"
Jul 17 13:33:58.251574 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251525    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94949073-5836-48f3-8abb-9e45b94ceca0" containerName="revoke-keys"
Jul 17 13:33:58.251574 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251531    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="94949073-5836-48f3-8abb-9e45b94ceca0" containerName="revoke-keys"
Jul 17 13:33:58.251574 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251537    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e5d7cb-f678-411b-b88d-1644a9f46bf1" containerName="revoke-keys"
Jul 17 13:33:58.251574 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251543    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e5d7cb-f678-411b-b88d-1644a9f46bf1" containerName="revoke-keys"
Jul 17 13:33:58.251574 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251549    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16679337-25ad-4a0f-bc6a-db2f1f7f643d" containerName="revoke-keys"
Jul 17 13:33:58.251574 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251553    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="16679337-25ad-4a0f-bc6a-db2f1f7f643d" containerName="revoke-keys"
Jul 17 13:33:58.251574 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251559    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16679337-25ad-4a0f-bc6a-db2f1f7f643d" containerName="revoke-keys"
Jul 17 13:33:58.251574 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251563    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="16679337-25ad-4a0f-bc6a-db2f1f7f643d" containerName="revoke-keys"
Jul 17 13:33:58.252069 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251620    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e5d7cb-f678-411b-b88d-1644a9f46bf1" containerName="revoke-keys"
Jul 17 13:33:58.252069 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251628    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e5d7cb-f678-411b-b88d-1644a9f46bf1" containerName="revoke-keys"
Jul 17 13:33:58.252069 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251633    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="16679337-25ad-4a0f-bc6a-db2f1f7f643d" containerName="revoke-keys"
Jul 17 13:33:58.252069 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251638    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="94949073-5836-48f3-8abb-9e45b94ceca0" containerName="revoke-keys"
Jul 17 13:33:58.252069 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251644    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="94949073-5836-48f3-8abb-9e45b94ceca0" containerName="revoke-keys"
Jul 17 13:33:58.252069 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251650    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="94949073-5836-48f3-8abb-9e45b94ceca0" containerName="revoke-keys"
Jul 17 13:33:58.252069 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251656    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="16679337-25ad-4a0f-bc6a-db2f1f7f643d" containerName="revoke-keys"
Jul 17 13:33:58.252069 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.251662    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e5d7cb-f678-411b-b88d-1644a9f46bf1" containerName="revoke-keys"
Jul 17 13:33:58.254598 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.254577    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:33:58.257176 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.257150    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8mj6w\"/\"openshift-service-ca.crt\""
Jul 17 13:33:58.258301 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.258282    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8mj6w\"/\"kube-root-ca.crt\""
Jul 17 13:33:58.258424 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.258304    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8mj6w\"/\"default-dockercfg-lsxbf\""
Jul 17 13:33:58.266163 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.266142    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"]
Jul 17 13:33:58.402745 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.402712    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/068b25c2-369e-4edb-82c5-7d641637d8dd-podres\") pod \"perf-node-gather-daemonset-5txrk\" (UID: \"068b25c2-369e-4edb-82c5-7d641637d8dd\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:33:58.402900 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.402782    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbcgj\" (UniqueName: \"kubernetes.io/projected/068b25c2-369e-4edb-82c5-7d641637d8dd-kube-api-access-fbcgj\") pod \"perf-node-gather-daemonset-5txrk\" (UID: \"068b25c2-369e-4edb-82c5-7d641637d8dd\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:33:58.402900 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.402854    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/068b25c2-369e-4edb-82c5-7d641637d8dd-sys\") pod \"perf-node-gather-daemonset-5txrk\" (UID: \"068b25c2-369e-4edb-82c5-7d641637d8dd\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:33:58.402900 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.402882    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/068b25c2-369e-4edb-82c5-7d641637d8dd-lib-modules\") pod \"perf-node-gather-daemonset-5txrk\" (UID: \"068b25c2-369e-4edb-82c5-7d641637d8dd\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:33:58.403054 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.402912    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/068b25c2-369e-4edb-82c5-7d641637d8dd-proc\") pod \"perf-node-gather-daemonset-5txrk\" (UID: \"068b25c2-369e-4edb-82c5-7d641637d8dd\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:33:58.503584 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.503516    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/068b25c2-369e-4edb-82c5-7d641637d8dd-sys\") pod \"perf-node-gather-daemonset-5txrk\" (UID: \"068b25c2-369e-4edb-82c5-7d641637d8dd\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:33:58.503584 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.503549    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/068b25c2-369e-4edb-82c5-7d641637d8dd-lib-modules\") pod \"perf-node-gather-daemonset-5txrk\" (UID: \"068b25c2-369e-4edb-82c5-7d641637d8dd\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:33:58.503584 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.503573    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/068b25c2-369e-4edb-82c5-7d641637d8dd-proc\") pod \"perf-node-gather-daemonset-5txrk\" (UID: \"068b25c2-369e-4edb-82c5-7d641637d8dd\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:33:58.503848 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.503629    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/068b25c2-369e-4edb-82c5-7d641637d8dd-podres\") pod \"perf-node-gather-daemonset-5txrk\" (UID: \"068b25c2-369e-4edb-82c5-7d641637d8dd\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:33:58.503848 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.503658    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbcgj\" (UniqueName: \"kubernetes.io/projected/068b25c2-369e-4edb-82c5-7d641637d8dd-kube-api-access-fbcgj\") pod \"perf-node-gather-daemonset-5txrk\" (UID: \"068b25c2-369e-4edb-82c5-7d641637d8dd\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:33:58.503848 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.503664    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/068b25c2-369e-4edb-82c5-7d641637d8dd-sys\") pod \"perf-node-gather-daemonset-5txrk\" (UID: \"068b25c2-369e-4edb-82c5-7d641637d8dd\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:33:58.503848 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.503740    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/068b25c2-369e-4edb-82c5-7d641637d8dd-proc\") pod \"perf-node-gather-daemonset-5txrk\" (UID: \"068b25c2-369e-4edb-82c5-7d641637d8dd\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:33:58.503848 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.503782    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/068b25c2-369e-4edb-82c5-7d641637d8dd-lib-modules\") pod \"perf-node-gather-daemonset-5txrk\" (UID: \"068b25c2-369e-4edb-82c5-7d641637d8dd\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:33:58.503848 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.503809    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/068b25c2-369e-4edb-82c5-7d641637d8dd-podres\") pod \"perf-node-gather-daemonset-5txrk\" (UID: \"068b25c2-369e-4edb-82c5-7d641637d8dd\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:33:58.511641 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.511620    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbcgj\" (UniqueName: \"kubernetes.io/projected/068b25c2-369e-4edb-82c5-7d641637d8dd-kube-api-access-fbcgj\") pod \"perf-node-gather-daemonset-5txrk\" (UID: \"068b25c2-369e-4edb-82c5-7d641637d8dd\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:33:58.565355 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.565328    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:33:58.697565 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.697542    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"]
Jul 17 13:33:58.702788 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:33:58.700430    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod068b25c2_369e_4edb_82c5_7d641637d8dd.slice/crio-0ec3bea126a2bf9cc96454e5cac30e51078c19b9fa530bff3b3a86aa1ba40ecc WatchSource:0}: Error finding container 0ec3bea126a2bf9cc96454e5cac30e51078c19b9fa530bff3b3a86aa1ba40ecc: Status 404 returned error can't find the container with id 0ec3bea126a2bf9cc96454e5cac30e51078c19b9fa530bff3b3a86aa1ba40ecc
Jul 17 13:33:58.740551 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:58.740526    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk" event={"ID":"068b25c2-369e-4edb-82c5-7d641637d8dd","Type":"ContainerStarted","Data":"0ec3bea126a2bf9cc96454e5cac30e51078c19b9fa530bff3b3a86aa1ba40ecc"}
Jul 17 13:33:59.111631 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:59.111552    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7b5cc587cc-j84vt_8c03aaeb-0c17-4472-8a65-6b168083cce1/console/0.log"
Jul 17 13:33:59.145736 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:59.145708    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-jvdrq_7fbc6f80-0452-475d-a501-deb3474f788a/download-server/0.log"
Jul 17 13:33:59.746374 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:59.746339    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk" event={"ID":"068b25c2-369e-4edb-82c5-7d641637d8dd","Type":"ContainerStarted","Data":"018a04b73a5a692eb664015d8447b6a7d8a2f7cecaf560f5ecc8a12ca1270505"}
Jul 17 13:33:59.746791 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:59.746441    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:33:59.762591 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:33:59.762547    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk" podStartSLOduration=1.7625338720000001 podStartE2EDuration="1.762533872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 13:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:33:59.762159431 +0000 UTC m=+4314.766206324" watchObservedRunningTime="2026-07-17 13:33:59.762533872 +0000 UTC m=+4314.766580753"
Jul 17 13:34:00.784415 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:00.784390    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mq9p4_668cc1ae-8c83-45b9-a875-b65bd76f80ea/dns/0.log"
Jul 17 13:34:00.803990 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:00.803966    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mq9p4_668cc1ae-8c83-45b9-a875-b65bd76f80ea/kube-rbac-proxy/0.log"
Jul 17 13:34:00.919446 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:00.919421    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lmgzk_460481eb-fc51-49b6-b8f7-7e892d0d0e64/dns-node-resolver/0.log"
Jul 17 13:34:01.528114 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:01.528087    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q7w4l_a2bedbc3-150b-4334-a07f-574108c2901a/node-ca/0.log"
Jul 17 13:34:02.596985 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:02.596950    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wh79n_4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb/discovery/0.log"
Jul 17 13:34:03.467498 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:03.467471    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4nsvg_e9ff568b-7a93-4a5d-9eac-b04e3b285830/serve-healthcheck-canary/0.log"
Jul 17 13:34:04.066639 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:04.066611    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ctcgp_70772ec3-2b0c-4134-9be0-039e0a2c0dc4/kube-rbac-proxy/0.log"
Jul 17 13:34:04.088576 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:04.088553    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ctcgp_70772ec3-2b0c-4134-9be0-039e0a2c0dc4/exporter/0.log"
Jul 17 13:34:04.109418 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:04.109395    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ctcgp_70772ec3-2b0c-4134-9be0-039e0a2c0dc4/extractor/0.log"
Jul 17 13:34:05.761376 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:05.761349    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-5txrk"
Jul 17 13:34:06.794560 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:06.794534    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d8fb4667b-ccn8z_1272b7db-2ffc-4f97-b60a-693a9b5749d6/manager/0.log"
Jul 17 13:34:06.813048 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:06.813022    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d8fb4667b-ccn8z_1272b7db-2ffc-4f97-b60a-693a9b5749d6/copy-manifests/0.log"
Jul 17 13:34:08.301062 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:08.301037    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-774b4c8dcd-v4v2x_1246e75c-64df-4548-9c61-4d8578615b17/manager/0.log"
Jul 17 13:34:08.948475 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:08.948447    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wh79n_4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb/discovery/0.log"
Jul 17 13:34:10.459422 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:10.459397    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wh79n_4ded48e5-a3fc-4fd4-b680-dfd7b27b27bb/discovery/0.log"
Jul 17 13:34:11.894228 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:11.894198    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-pm624"]
Jul 17 13:34:11.894588 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:11.894508    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94949073-5836-48f3-8abb-9e45b94ceca0" containerName="revoke-keys"
Jul 17 13:34:11.894588 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:11.894518    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="94949073-5836-48f3-8abb-9e45b94ceca0" containerName="revoke-keys"
Jul 17 13:34:11.894588 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:11.894526    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e5d7cb-f678-411b-b88d-1644a9f46bf1" containerName="revoke-keys"
Jul 17 13:34:11.894588 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:11.894532    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e5d7cb-f678-411b-b88d-1644a9f46bf1" containerName="revoke-keys"
Jul 17 13:34:11.894588 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:11.894590    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="16679337-25ad-4a0f-bc6a-db2f1f7f643d" containerName="revoke-keys"
Jul 17 13:34:11.908320 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:11.908294    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-pm624"
Jul 17 13:34:11.909047 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:11.909019    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-pm624"]
Jul 17 13:34:11.910777 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:11.910739    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:34:11.910895 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:11.910774    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 13:34:11.997300 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:11.997273    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37c02151-b33e-4f5f-a38e-76eec4c6b950-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-pm624\" (UID: \"37c02151-b33e-4f5f-a38e-76eec4c6b950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-pm624"
Jul 17 13:34:12.097745 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:12.097720    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37c02151-b33e-4f5f-a38e-76eec4c6b950-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-pm624\" (UID: \"37c02151-b33e-4f5f-a38e-76eec4c6b950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-pm624"
Jul 17 13:34:12.098257 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:12.098238    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37c02151-b33e-4f5f-a38e-76eec4c6b950-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-pm624\" (UID: \"37c02151-b33e-4f5f-a38e-76eec4c6b950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-pm624"
Jul 17 13:34:12.218699 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:12.218650    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-pm624"
Jul 17 13:34:12.356652 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:12.356616    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-pm624"]
Jul 17 13:34:12.363426 ip-10-0-131-210 kubenswrapper[2569]: W0717 13:34:12.363398    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c02151_b33e_4f5f_a38e_76eec4c6b950.slice/crio-af72f67bc4eeadeb49988707a4d7313d0121d348c3667c235dc66f421db26fc7 WatchSource:0}: Error finding container af72f67bc4eeadeb49988707a4d7313d0121d348c3667c235dc66f421db26fc7: Status 404 returned error can't find the container with id af72f67bc4eeadeb49988707a4d7313d0121d348c3667c235dc66f421db26fc7
Jul 17 13:34:12.805502 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:12.805440    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-pm624" event={"ID":"37c02151-b33e-4f5f-a38e-76eec4c6b950","Type":"ContainerStarted","Data":"f81c8e12f13cb030e35e0a3d466e867f13ea4fb648f88a715202c303c7a749c9"}
Jul 17 13:34:12.805502 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:12.805497    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-pm624" event={"ID":"37c02151-b33e-4f5f-a38e-76eec4c6b950","Type":"ContainerStarted","Data":"af72f67bc4eeadeb49988707a4d7313d0121d348c3667c235dc66f421db26fc7"}
Jul 17 13:34:12.829768 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:12.829704    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-pm624" podStartSLOduration=1.829688612 podStartE2EDuration="1.829688612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:34:12.82777043 +0000 UTC m=+4327.831817302" watchObservedRunningTime="2026-07-17 13:34:12.829688612 +0000 UTC m=+4327.833735494"
Jul 17 13:34:13.437308 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:13.437280    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-2g2f7_4bbbe48c-ae26-4550-af9b-77f6efdbc830/migrator/0.log"
Jul 17 13:34:13.459936 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:13.459914    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-2g2f7_4bbbe48c-ae26-4550-af9b-77f6efdbc830/graceful-termination/0.log"
Jul 17 13:34:13.812231 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:13.812203    2569 generic.go:356] "Generic (PLEG): container finished" podID="37c02151-b33e-4f5f-a38e-76eec4c6b950" containerID="f81c8e12f13cb030e35e0a3d466e867f13ea4fb648f88a715202c303c7a749c9" exitCode=6
Jul 17 13:34:13.812384 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:13.812284    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-pm624" event={"ID":"37c02151-b33e-4f5f-a38e-76eec4c6b950","Type":"ContainerDied","Data":"f81c8e12f13cb030e35e0a3d466e867f13ea4fb648f88a715202c303c7a749c9"}
Jul 17 13:34:13.812454 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:13.812426    2569 scope.go:117] "RemoveContainer" containerID="f81c8e12f13cb030e35e0a3d466e867f13ea4fb648f88a715202c303c7a749c9"
Jul 17 13:34:14.818977 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:14.818948    2569 generic.go:356] "Generic (PLEG): container finished" podID="37c02151-b33e-4f5f-a38e-76eec4c6b950" containerID="c4fa20d90536cd3d40815242bb402f9f54b483784a4016fa8675a8fda71477d7" exitCode=6
Jul 17 13:34:14.819352 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:14.818991    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-pm624" event={"ID":"37c02151-b33e-4f5f-a38e-76eec4c6b950","Type":"ContainerDied","Data":"c4fa20d90536cd3d40815242bb402f9f54b483784a4016fa8675a8fda71477d7"}
Jul 17 13:34:14.819352 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:14.819043    2569 scope.go:117] "RemoveContainer" containerID="f81c8e12f13cb030e35e0a3d466e867f13ea4fb648f88a715202c303c7a749c9"
Jul 17 13:34:14.819352 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:14.819221    2569 scope.go:117] "RemoveContainer" containerID="c4fa20d90536cd3d40815242bb402f9f54b483784a4016fa8675a8fda71477d7"
Jul 17 13:34:14.819512 ip-10-0-131-210 kubenswrapper[2569]: E0717 13:34:14.819409    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-pm624_odh-ai-gateway-infra(37c02151-b33e-4f5f-a38e-76eec4c6b950)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-pm624" podUID="37c02151-b33e-4f5f-a38e-76eec4c6b950"
Jul 17 13:34:15.129307 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:15.129250    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whc8l_2680a751-e4a6-438e-bcd9-51f8ab07fa3b/kube-multus-additional-cni-plugins/0.log"
Jul 17 13:34:15.152838 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:15.152815    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whc8l_2680a751-e4a6-438e-bcd9-51f8ab07fa3b/egress-router-binary-copy/0.log"
Jul 17 13:34:15.175242 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:15.175223    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whc8l_2680a751-e4a6-438e-bcd9-51f8ab07fa3b/cni-plugins/0.log"
Jul 17 13:34:15.197761 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:15.197739    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whc8l_2680a751-e4a6-438e-bcd9-51f8ab07fa3b/bond-cni-plugin/0.log"
Jul 17 13:34:15.218735 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:15.218716    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whc8l_2680a751-e4a6-438e-bcd9-51f8ab07fa3b/routeoverride-cni/0.log"
Jul 17 13:34:15.238856 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:15.238835    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whc8l_2680a751-e4a6-438e-bcd9-51f8ab07fa3b/whereabouts-cni-bincopy/0.log"
Jul 17 13:34:15.258011 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:15.257987    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whc8l_2680a751-e4a6-438e-bcd9-51f8ab07fa3b/whereabouts-cni/0.log"
Jul 17 13:34:15.300882 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:15.300863    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-g4c6m_80c7a3a0-1833-46d5-bd61-7843b21bd530/kube-multus/0.log"
Jul 17 13:34:15.396641 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:15.396571    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4nbwg_966fc2f5-1448-4a14-8b27-21e714628e1a/network-metrics-daemon/0.log"
Jul 17 13:34:15.413054 ip-10-0-131-210 kubenswrapper[2569]: I0717 13:34:15.413039    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4nbwg_966fc2f5-1448-4a14-8b27-21e714628e1a/kube-rbac-proxy/0.log"